System and method for guiding a user to a goal in a user interface

ABSTRACT

A system for integrating a telephone system and a computing system includes an interactive voice response (IVR) platform configured to: obtain a command based on an audio input from a user, determine a web application that corresponds to the command, determine a goal in the web application associated with the command, and obtain information indicating a shortest user interface path to the goal. The system includes a cobrowse client configured to receive a document object model (DOM) of a current state of the web application from a cobrowse session. The IVR platform is configured to, based on the DOM, determine a next user interface action along the shortest user interface path and generate a voice prompt for the user based on the next user interface action. The cobrowse client is configured to receive an updated DOM in response to execution by the user of the next user interface action.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present disclosure is a continuation of U.S. patent application Ser.No. 16/196,404 filed on 20 Nov. 2018. The entire disclosure of theapplication referenced above is incorporated herein by reference.

FIELD

The present disclosure relates to interactive voice response (IVR)systems and more particularly to IVR systems associated withcorresponding web-based systems.

BACKGROUND

Currently, entities such as companies offer customers a variety ofmethods to access information—for example, a status of an order that thecustomer placed. The customer may call the company in an attempt toobtain the information. Alternatively, the customer may access a webpage associated with the company to obtain the information. The customermay need to navigate a user interface (UI) of an application to accessthe information. As an example, the customer may need to log in to a webapplication with a username and password before they are able to accessthe information. Many customers may be unable to properly navigate theUI—for example, to log in to the web application and navigate to thepertinent screen—and, therefore, may opt to call the company to obtainthe information.

The background description provided here is for the purpose of generallypresenting the context of the disclosure. Work of the presently namedinventors, to the extent it is described in this background section, aswell as aspects of the description that may not otherwise qualify asprior art at the time of filing, are neither expressly nor impliedlyadmitted as prior art against the present disclosure.

SUMMARY

A computer system for testing a user interface (UI) is disclosed. Thecomputer system includes a test creation module configured to obtaintesting parameters. The testing parameters include (i) a location atwhich the UI can be accessed and (ii) criteria for a goal associatedwith the UI. The computer system also includes a test execution moduleconfigured to (i) obtain a state of the UI based on the location and(ii) set a current position to a predetermined location within the UI.The system further includes an analysis module configured to (i) analyzea designated state of the UI, (ii) in response to determining that thedesignated state satisfies the criteria for the goal, output a successindicator, and (iii) in response to determining that the designatedstate of the UI does not satisfy the criteria for the goal, determine aset of possible actions based on UI elements within the designated stateand set the outcome to the set of possible actions. The test executionmodule is configured to provide a state of the UI to the analysis moduleand receive the output from the analysis module. The test executionmodule is also configured to, in response to the received output beingthe set of possible actions: select an action associated with a first UIelement from the set of possible actions; execute the selected action;identify a point of the first UI element; update a test distance basedon (i) coordinates of the point and (ii) coordinates of the currentposition; set the current position to the point; and supplement testdata with (i) the selected action and (ii) the set of possible actions.The test execution module is further configured to (i) in response tothe received output being the success indicator, store the test distanceand the test data in a collection of completed tests and (ii) determinea shortest path to the goal in the UI based on the collection ofcompleted tests.

In other features, the testing parameters include a number of permittedtests. The test execution module is configured to increment a counter inresponse to the received output being the success indicator. Determiningthe shortest path includes selecting, in response to determining that avalue of the counter is greater than or equal to the number of permittedtests, a completed test of the collection of completed tests with theshortest test distance.

In yet other features, the point is a closest point of the first UIelement to the current position and updating the test distance includes(i) determining a distance between the coordinates of the currentposition and the coordinates of the point and (ii) adding the determineddistance to the test distance.

In other features, the analysis module is configured to, for each actionin the set of possible actions, (i) determine a probability thatperforming the action will result in the goal and (ii) store thedetermined probability in the set of possible actions. Selecting theaction from the set of possible includes selecting the action from theset of possible actions based on probabilities stored in the set ofpossible actions.

In further features, the computer system includes a neural networkmodule configured to operate a plurality of neural networks. Theanalysis module is configured to selectively use at least one neuralnetwork of the plurality of neural networks to determine the probabilitythat performing the action will result in the goal.

In yet further features, the plurality of neural networks includes atleast one of a long short-term memory neural network and a convolutionalneural network.

In other features, the computer system includes a training moduleconfigured to, in response to the received output being the successindicator: train the plurality of neural networks using the test data;determine a performance of the plurality of neural networks aftertraining based on the probabilities stored in the collection ofcompleted tests; and in response to determining that a value of theperformance of the plurality of neural networks after training isgreater than a predetermined value, set a status of the plurality ofneural networks to trained.

In further features, the computer system includes a Monte Carlo moduleconfigured to perform a Monte Carlo simulation to generate a randomvalue and a reweighting module configured to (i) in response to thestatus of the plurality of neural networks being trained, update eachprobability stored in the test data based on an output of at least oneneural network of the plurality of neural networks and (ii) in responseto the status of the plurality of neural networks not being trained,update each probability stored in the test data with a random valuegenerated by the Monte Carlo module.

A method for testing a user interface (UI) is disclosed. The methodincludes obtaining testing parameters. The testing parameters include(i) a location at which the UI can be accessed and (ii) criteria for agoal associated with the UI. The method also includes: obtaining a stateof the UI based on the location; setting a current position to apredetermined location within the UI; analyzing a designated state ofthe UI; in response to determining that the designated state satisfiesthe criteria for the goal, setting an output to a success indicator; andin response to determining that the designated state does not satisfythe criteria for the goal, (i) determining a set of possible actionsbased on UI elements within the designated state and (ii) setting theoutput to the set of possible actions. The method further includes, inresponse to the output being the set of possible actions: selecting anaction associated with a first UI element from the set of possibleactions; executing the selected action; identifying a point of the firstUI element; updating a test distance based on (i) coordinates of thepoint and (ii) coordinates of the current position; setting the currentposition to the point; and supplement test data with (i) the selectedaction and (ii) the set of possible actions. The method also includes,(i) in response to the output being the success indicator, storing thetest distance and the test data in a collection of completed tests and(ii) determining a shortest path to the goal in the UI based on thecollection of completed tests.

In other features, the testing parameters include a number of permittedtests. The method includes incrementing a counter in response to theoutput being the success indicator. Determining the shortest pathincludes, in response to determining that a value of the counter isgreater than or equal to the number of permitted tests, selecting acompleted test of the collection of completed tests with the shortesttest distance.

In yet other features, the point is a closest point of the first UIelement to the current position. Updating the test distance includes (i)determining a distance between the coordinates of the current positionand the coordinates of the point and (ii) adding the determined distanceto the test distance.

In other features, the method includes, for each action in the set ofpossible actions, (i) determining a probability that performing theaction will result in the goal and (ii) storing the determinedprobability in the set of possible actions. Selecting the action fromthe set of possible actions includes selecting the action from the setof possible actions based on probabilities stored in the set of possibleactions.

In further features, the method includes, in response to the outputbeing the success indicator: training a plurality of neural networksusing the test data; determining a performance of the plurality ofneural networks after training based on the probabilities stored in thecollection of completed tests; and in response to determining that avalue of the performance of the plurality of neural networks aftertraining is greater than a predetermined value, setting a status of theplurality of neural networks to trained.

In yet further features, the method includes, in response to the statusof the plurality of neural networks being trained, updating eachprobability stored in the test data based on an output of at least oneneural network of the plurality of neural networks and in response tothe status of the plurality of neural networks not being trained,updating each probability stored in the test data with a random value.

A non-transitory computer-readable medium storing processor-executableinstructions is disclosed. The instructions include obtaining testingparameters. The testing parameters include (i) a location at which auser interface (UI) can be accessed and (ii) criteria for a goalassociated with the UI. The instructions also include: obtaining a stateof the UI based on the location; setting a current position to apredetermined location within the UI; analyzing a designated state ofthe UI; in response to determining that the designated state satisfiesthe criteria for the goal, setting an output to a success indicator; andin response to determining that the designated state does not satisfythe criteria for the goal, (i) determining a set of possible actionsbased on UI elements within the designated state and (ii) setting theoutput to the set of possible actions. The instructions further include,in response to the output being the set of possible actions: selectingan action associated with a first UI element from the set of possibleactions; executing the selected action; identifying a point of the firstUI element; updating a test distance based on (i) coordinates of thepoint and (ii) coordinates of the current position; setting the currentposition to the point; and supplementing test data with (i) the selectedaction and (ii) the set of possible actions. The instructions alsoinclude (i) in response to the output being the success indicator,storing the test distance and the test data in a collection of completedtests and (ii) determining a shortest path to the goal in the UI basedon the collection of completed tests.

In other features, the testing parameters include a number of permittedtests. The instructions include incrementing a counter in response tothe output being the success indicator. Determining the shortest pathincludes, in response to determining that a value of the counter isgreater than or equal to the number of permitted tests, selecting acompleted test of the collection of completed tests with the shortesttest distance.

In yet other features, the point is a closest point of the first UIelement to the current position. Updating the test distance includes (i)determining a distance between the coordinates of the current positionand the coordinates of the point and (ii) adding the determined distanceto the test distance.

In other features, the instructions include, for each action in the setof possible actions, (i) determining a probability that performing theaction will result in the goal and (ii) storing the determinedprobability in the set of possible actions. Selecting the action fromthe set of possible includes selecting the action from the set ofpossible actions based on probabilities stored in the set of possibleactions.

In further features, the instructions include, in response to the outputbeing the success indicator: training a plurality of neural networksusing the test data; determining a performance of the plurality ofneural networks after training based on the probabilities stored in thecollection of completed tests; and setting, in response to determiningthat a value of the performance of the plurality of neural networksafter training is greater than a predetermined value, a status of theplurality of neural networks to trained.

In yet further features, the instructions include (i) in response to thestatus of the plurality of neural networks being trained, updating eachprobability stored in the test data based on an output of at least oneneural network of the plurality of neural networks and (ii) in responseto the status of the plurality of neural networks not being trained,updating each probability stored in the test data with a random value.

A computer system for improving a user interface (UI) is disclosed. Thecomputer system includes a reinforcement module configured to obtain (i)path information indicating a shortest path to a goal in the UI and (ii)a set of user interaction experiments associated with seeking the goal.Each experiment in the set of user interaction experiments includes usertracking data, the path information includes a plurality of steps, andeach step of the plurality of steps is associated with a page of the UI.The computer system also includes a distance module configured to, foreach page of the UI, determine a reinforcement distance for the pagebased on the set of user interaction experiments and a step analysismodule configured to, for each step of the plurality of steps, determinea count of times that the user tracking data of the set of userinteraction experiments indicates a deviation from the step. Thedetermination is based on the determined reinforcement distance for thepage of the UI associated with the step. The reinforcement module isconfigured to, for each step of the plurality of steps, (i) determinewhether the step requires reinforcement based on the count and (ii) inresponse to determining that the step requires reinforcement, generate areinforcement for the step. Generating the reinforcement includes atleast one of generating a prompt for an action associated with the stepand altering a UI element associated with the step.

In other features, determining a reinforcement distance for a pageincludes determining at least one distance between user tracking data ofthe set of user interaction experiments and the shortest path.

In yet other features, determining a reinforcement distance for a pageincludes determining (i) a median successful distance for the page basedon the user tracking data of a first subset of user interactionexperiments that successfully reached the goal and (ii) a median failuredistance for the page based on the user tracking data of a second subsetof user interaction experiments.

In further features, the distance module is configured to, for each pageof the UI, generate at least one random distance that is less than themedian successful distance for the page and greater than the medianfailure distance for the page. The computer system includes a predictionmodule configured to, for each sample in user tracking data of anexperiment in the set of user interaction experiments, predict anoutcome of the experiment based on (i) a generated random distance forthe page associated with the sample, (ii) coordinates of the sample, and(iii) coordinates of a point along the shortest path and compare thepredicted outcome to a stored outcome of the experiment. The predictionmodule is also configured to output a result of the comparisons.

In other features, predicting an outcome of the experiment includes (i)calculating a distance between the coordinates of the sample and thecoordinates of the point along the shortest path and (ii) determiningthe predicted outcome based on a comparison of the determined distanceand the random distance for the page associated with the sample.Comparing the predicted outcome to the stored outcome includes, inresponse to the predicted outcome matching the stored outcome,incrementing a successful prediction count. Outputting the result of thecomparisons includes outputting the successful prediction count.

In yet other features, the distance module is configured to, for eachrandomly generated distance, generate a fitness value based at least onthe successful prediction count outputted by the prediction module. Thecomputer system includes an analysis module configured to perform acluster analysis of the generated fitness values.

In further features, the distance module is configured to, for each pageof the UI, in response to the cluster analysis of the generated fitnessvalues identifying a single cluster, set the randomly generated distanceassociated with the page with the highest fitness value as thereinforcement distance for the page.

A method for improving a user interface (UI) is disclosed. The methodincludes obtaining (i) path information indicating a shortest path to agoal in the UI and (ii) a set of user interaction experiments associatedwith seeking the goal. Each experiment in the set of user interactionexperiments includes user tracking data, the path information includes aplurality of steps, and each step of the plurality of steps isassociated with a page of the UI. The method also includes for each pageof the UI, determining a reinforcement distance for the page based onthe set of user interaction experiments and for each step of theplurality of steps, determining a count of times that the user trackingdata of the set of user interaction experiments indicates a deviationfrom the step. The determination is based on the determinedreinforcement distance for the page of the UI associated with the step.The method also includes determining whether the step requiresreinforcement based on the count and in response to determining that thestep requires reinforcement, generating a reinforcement for the step.Generating the reinforcement includes at least one of generating aprompt for an action associated with the step and altering a UI elementassociated with the step.

In other features, determining the reinforcement distance for a pageincludes determining at least one distance between user tracking data ofthe set of user interaction experiments and the shortest path.

In yet other features, determining the reinforcement distance for a pageincludes determining (i) a median successful distance for the page basedon the user tracking data of a first subset of user interactionexperiments that successfully reached the goal and (ii) a median failuredistance for the page based on the user tracking data of a second subsetof user interaction experiments.

In further features, determining the reinforcement distance for a pageincludes generating at least one random distance that is less than themedian successful distance for the page and greater than the medianfailure distance for the page. The method also includes, for each samplein user tracking data of an experiment in the set of user interactionexperiments, predicting an outcome of the experiment based on (i) agenerated random distance for the page associated with the sample, (ii)coordinates of the sample, and (iii) coordinates of a point along theshortest path and comparing the predicted outcome to a stored outcome ofthe experiment.

In other features, predicting the outcome of the experiment includescalculating a distance between the coordinates of the sample and thecoordinates of the point along the shortest path and determining thepredicted outcome based on a comparison of the determined distance andthe random distance for the page associated with the sample. Comparingthe predicted outcome to the stored outcome includes, in response to thepredicted outcome matching the stored outcome, incrementing a successfulprediction count.

In yet other features, determining the reinforcement distance for a pageincludes, for each randomly generated distance, generating a fitnessvalue based on at least the successful prediction count associated withrandomly generated distance and performing a cluster analysis of thegenerated fitness values.

In further features, determining the reinforcement distance for a pageincludes, in response to the cluster analysis of the generated fitnessvalues identifying a single cluster, setting the randomly generateddistance associated with the page with the highest fitness value as thereinforcement distance for the page.

A non-transitory computer-readable medium storing processor-executableinstructions is disclosed. The instructions include obtaining (i) pathinformation indicating a shortest path to a goal in a user interface(UI) and (ii) a set of user interaction experiments associated withseeking the goal. Each experiment in the set of user interactionexperiments includes user tracking data, the path information includes aplurality of steps, and each step of the plurality of steps isassociated with a page of the UI. The instructions also include, foreach page of the UI, determining a reinforcement distance for the pagebased on the set of user interaction experiments and for each step ofthe plurality of steps, determining a count of times that the usertracking data of the set of user interaction experiments indicates adeviation from the step. The determination is based on the determinedreinforcement distance for the page of the UI associated with the step.The instructions further include, for each step of the plurality ofsteps, determining whether the step requires reinforcement based on thecount and in response to determining that the step requiresreinforcement, generating a reinforcement for the step. Generating thereinforcement includes at least one of generating a prompt for an actionassociated with the step and altering a UI element associated with thestep.

In other features, determining the reinforcement distance for a pageincludes determining (i) a median successful distance for the page basedon the user tracking data of a first subset of user interactionexperiments that successfully reached the goal and (ii) a median failuredistance for the page based on the user tracking data of a second subsetof user interaction experiments.

In yet other features, determining the reinforcement distance for a pageincludes generating at least one random distance that is less than themedian successful distance for the page and greater than the medianfailure distance for the page. The instructions also include, for eachsample in user tracking data of an experiment in the set of userinteraction experiments, predicting an outcome of the experiment basedon (i) a generated random distance for the page associated with thesample, (ii) coordinates of the sample, and (iii) coordinates of a pointalong the shortest path. The instructions further include comparing thepredicted outcome to a stored outcome of the experiment.

In further features, predicting the outcome of the experiment includes(i) calculating a distance between the coordinates of the sample and thecoordinates of the point along the shortest path and (ii) determiningthe predicted outcome based on a comparison of the determined distanceand the random distance for the page associated with the sample.Comparing the predicted outcome to the stored outcome includes, inresponse to the predicted outcome matching the stored outcome,incrementing a successful prediction count.

In other features, determining the reinforcement distance for a pageincludes, for each randomly generated distance, (i) generating a fitnessvalue based on at least the successful prediction count associated withthe randomly generated distance and (ii) performing a cluster analysisof the generated fitness values.

In yet other features, determining the reinforcement distance for a pageincludes, in response to the cluster analysis of the generated fitnessvalues identifying a single cluster, setting the randomly generateddistance associated with the page with the highest fitness value as thereinforcement distance for the page.

A system for integrating a telephone system and a computing system isdisclosed. The system includes an interactive voice response (IVR)platform configured to obtain a computer-readable command based on anaudio input from a user and in response to obtaining thecomputer-readable command, (i) determine a web application thatcorresponds to the computer-readable command, (ii) determine a goal inthe web application associated with the computer-readable command, and(iii) obtain information indicating a shortest user interface path tothe goal in the web application. The system also includes a cobrowseclient configured to receive a document object model (DOM) of a currentstate of the web application from a cobrowse session for a web serverhosting the web application. The IVR platform is configured to, based onthe DOM from the cobrowse client, determine a next user interface actionalong the shortest user interface path and generate a voice prompt forthe user based on the next user interface action. The cobrowse client isconfigured to receive an updated DOM in response to execution by theuser of the next user interface action.

In other features, the IVR platform is configured to connect to thecobrowse client using a headless browser.

In yet other features, obtaining the audio input from the user includesobtaining the audio input via a telephone of the user.

In other features, the IVR platform is configured to obtain a cobrowsesession identifier from the user, transmit the cobrowse sessionidentifier to the cobrowse session, and receive the DOM of the currentstate in response to transmitting the cobrowse session identifier.

In further features, obtaining the cobrowse session identifier includesgenerating a voice instruction for the user that requests the user to(i) initiate the cobrowse session and (ii) provide the cobrowse sessionidentifier to the IVR platform.

In other features, the IVR platform is configured to, in response toexpiration of a predetermined time period subsequent to generating thevoice prompt during which no updated DOM is received, (i) determine areinforcement based on the next user interface action along the shortestuser interface path and (ii) generate an audio reinforcement prompt forthe user based on the determined reinforcement.

In further features, the audio reinforcement prompt specifies a locationof a user interface element associated with the next user interfaceaction.

A method for integrating a telephone system and a computer system isdisclosed. The method includes: obtaining a computer-readable commandbased on an audio input from a user; determining a web application thatcorresponds to the computer-readable command; determining a goal in theweb application associated with the computer-readable command; obtaininginformation indicating a shortest user interface path to the goal in theweb application; receiving a document object model (DOM) of a currentstate of the web application from a cobrowse session for a web serverhosting the web application; and determining a next user interfaceaction along the shortest user interface path. The method also includesgenerating a voice prompt for the user based on the next user interfaceaction and receiving an updated DOM in response to execution by the userof the next user interface action.

In other features, receiving the DOM of the current state of the webapplication includes accessing a cobrowse client via a headless browser.

In yet other features, obtaining the audio input from the user includesobtaining the audio input via a telephone of the user.

In other features, the method includes obtaining a cobrowse sessionidentifier from the user; transmitting the cobrowse session identifierto the cobrowse session; and receiving the DOM of the current stateincludes receiving the DOM in response to transmitting the cobrowsesession identifier.

In further features, obtaining the cobrowse session identifier includesgenerating a voice instruction for the user that requests the user to(i) initiate the cobrowse session and (ii) provide the cobrowse sessionidentifier via a PSTN.

In other features, the method includes, in response to expiration of apredetermined time period subsequent to generating the voice promptduring which no updated DOM is received, (i) determining a reinforcementbased on the next user interface action along the shortest userinterface path and (ii) generating an audio reinforcement prompt for theuser based on the determined reinforcement.

In further features, the audio reinforcement prompt specifies a locationof a user interface element associated with the next user interfaceaction.

A non-transitory computer-readable medium storing processor-executableinstructions is disclosed. The instructions include: obtaining acomputer-readable command based on an audio input from a user;determining a web application that corresponds to the computer-readablecommand; determining a goal in the web application associated with thecomputer-readable command; obtaining information indicating a shortestuser interface path to the goal in the web application; receiving adocument object model (DOM) of a current state of the web applicationfrom a cobrowse session for a web server hosting the web application;and determining a next user interface action along the shortest userinterface path. The instructions also include generating a voice promptfor the user based on the next user interface action and receiving anupdated DOM in response to execution by the user of the next userinterface action.

In other features, obtaining the audio input from the user includesobtaining the audio input via a telephone of the user.

In yet other features, the instructions includes: obtaining a cobrowsesession identifier from the user; transmitting the cobrowse sessionidentifier to the cobrowse session; and receiving the DOM of the currentstate includes receiving the DOM in response to transmitting thecobrowse session identifier.

In further features, obtaining the cobrowse session identifier includesgenerating a voice instruction for the user that requests the user to(i) initiate the cobrowse session and (ii) provide the cobrowse sessionidentifier via a PSTN.

In other features, the instructions include, in response to expirationof a predetermined time period subsequent to generating the voice promptduring which no updated DOM is received, (i) determining a reinforcementbased on the next user interface action along the shortest userinterface path and (ii) generating an audio reinforcement prompt for theuser based on the determined reinforcement.

In further features, the audio reinforcement prompt specifies a locationof a user interface element associated with the next user interfaceaction.

Further areas of applicability of the present disclosure will becomeapparent from the detailed description, the claims, and the drawings.The detailed description and specific examples are intended for purposesof illustration only and are not intended to limit the scope of thedisclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will become more fully understood from thedetailed description and the accompanying drawings.

FIG. 1 is a functional block diagram of an example system for testing auser interface (UI).

FIG. 2 is a functional block diagram of an example long short-termmemory neural network.

FIG. 3 is a functional block diagram of an example link neural network.

FIG. 4 is a functional block diagram of an example submit neuralnetwork.

FIG. 5A is a functional block diagram of an example data choice neuralnetwork.

FIG. 5B is a functional block diagram of an example data match neuralnetwork.

FIG. 6 is a functional block diagram of an example action weight neuralnetwork.

FIGS. 7A and 7B are a flowchart depicting an example method ofdetermining a shortest path to a given goal in a UI.

FIGS. 8A-8C are a flowchart depicting an example method of analyzing astate of a UI.

FIGS. 9A and 9B are a flowchart depicting an example method of trainingneural networks using completed shortest path tests.

FIGS. 10A-10C are a flowchart depicting an example method of reweightingactions in stored states.

FIG. 11 is a flowchart depicting an example method of building a new UIsession based on a selected shortest path action.

FIGS. 12A-12H are graphical representations of determining a shortestpath to a goal in a specific example UI.

FIG. 13 is a flowchart depicting an example method of enhancing a UIbased on eye tracking experiments.

FIG. 14 is a flowchart depicting an example method of determining areinforcement distance for each URL for a web-based UI.

FIG. 15 is a flowchart depicting an example method for determiningdistances between eye tracking experiments and points along a loadedshortest path.

FIG. 16 is a flowchart depicting an example method for determining thedistances between tracking data and points along a loaded shortest pathfor a page of the UI.

FIG. 17 is a flowchart depicting an example method of determining themedian successful distance for each URL in a set of eye trackingexperiments based on data stored in the eye tracking experiments.

FIG. 18 is a flowchart depicting an example method of determining themedian failure distance for each URL in a set of eye trackingexperiments based on data stored in the eye tracking experiments.

FIG. 19 is a flowchart depicting an example method for comparingpredictions of outcomes of eye tracking experiments with previouslystored results.

FIG. 20 is a flowchart depicting an example method for comparing aprediction of an outcome of an eye tracking experiment with previouslystored results.

FIG. 21 is a flowchart depicting an example method for determining thesteps of a shortest path to a goal in a UI that may requirereinforcement.

FIG. 22 is a flowchart depicting an example method for determining stepsof a shortest path associated with a page of an eye tracking experimentthat may need reinforcement.

FIGS. 23A-23E are visual representations of an example process ofgenerating an eye tracking vector and a shortest path vector for a pageof a UI.

FIG. 24 is a functional block diagram of an example customer migrationsystem.

FIG. 25 is a sequence diagram that represents an example initiation of acobrowse session by the customer migration system.

FIG. 26 is a sequence diagram that represents an example cobrowsesession of the customer migration system.

FIGS. 27A-27D are depictions of an example cobrowse session of acustomer migration system, according to the principles of the presentdisclosure.

FIG. 28 is a functional block diagram of an example system including ahigh-volume pharmacy.

FIG. 29 is a functional block diagram of an example pharmacy fulfillmentdevice, which may be deployed within the system of FIG. 28.

FIG. 30 is a functional block diagram of an example order processingdevice, which may be deployed within the system of FIG. 28.

In the drawings, reference numbers may be reused to identify similarand/or identical elements.

DETAILED DESCRIPTION

Introduction

A user interface (UI) provides access to information and may permitvarious actions to be performed. However, a user may need to performcertain actions in the UI before the user is able to access informationof interest. As an example, a user may need to first log in with ausername and password. Then, before they can access the information,they may need to navigate to a certain screen of the UI and select aparticular button. Many users may not be able to efficiently navigatethe UI to obtain the information.

In various implementations, a shortest path to a goal in a UI—such aslogging in and obtaining an order status—may be automaticallydetermined. The length of the path may be measured using a variety ofmetrics, such as the number of inputs required, the total movement ofthe mouse, the number of keystrokes or mouse click events, etc. Invarious implementations, the length of the path may be assessed based ontotal Euclidean distance of mouse travel—that is, the sum of linearmovements to each UI element where interaction is necessary (forexample, from a starting point to a textbox, from the textbox to asubmit button, and from the location of the submit button to a link thatis to be selected).

The shortest path may be used to enhance the UI. For example, empiricaldata (such as eye or mouse movement) measured while users areinteracting with the UI may be used to determine when users of the UIare more likely to deviate from the determined shortest path. The UI maybe enhanced based on the determined deviations—for example, UI elementsassociated with the deviations may be changed, prompts related to theshortest path may be generated, and/or elements of the UI can behighlighted to aid users in successfully navigating the UI.

These UI enhancements may also be used by a customer migration system toautomatically guide a user who has called an interactive voice response(IVR) platform (such as a customer service number) to a particular stateof a UI. For example, the customer migration system may assist the userin successfully logging in to a web application and navigating to astate of the web site where the user can obtain the desired information.The customer migration system may instruct the user to access a web pageand then generate audio and visual prompts based on the UI enhancementsto guide the user to perform actions in the UI that correspond to thedetermined shortest path to the goal.

FIG. 1 is a block diagram an of example implementation of a system 100for testing a UI. The system 100 includes a UI testing and enhancementdevice 110, a storage device 112, and an operator device 114. The UItesting and enhancement device 110 may include a shortest path module120, a UI enhancement module 122, an artificial intelligence (AI) module124, and a headless browser 126. The storage device 112 may includenonvolatile storage in communication with the shortest path module 120and the UI enhancement module 122.

The shortest path module 120 determines a shortest path to a goal in aUI. For example, the shortest path module 120 may determine the leastnumber of steps to log in to a web application. A step consists of anaction taken on a page of the UI—for example, clicking on a link orentering data into a text entry field. Execution of a step may changethe state of the UI—that is, transitioning to a different screen orstate of the UI, or changing the current state or screen so that more ordifferent information or choices are available.

The shortest path module 120 includes a test creation module 128 and atest execution module 130. The test execution module 130 includes ananalysis module 132, a training module 134, a reweighting module 136,and a session creation module 138.

The AI module 124 includes a Monte Carlo module 142, a neural networkmodule 144, and a natural language recognition module 146. The MonteCarlo module 142 performs a Monte Carlo simulation to generatepseudorandom values. The neural network module 144 may include a linkneural network 148, a submit neural network 150, a data match neuralnetwork 152, a data choice neural network 154, and an action weightneural network 156, each of which is described in more detail below.

The link neural network 148, the submit neural network 150, the datachoice neural network 154, and the action weight neural network 156 eachgenerate a probability that a given input will result in reaching thegoal. For example, the data match neural network 152 generates aprobability that a piece of data matches a data entry field—such as theprobability that a particular text string matches a text entry field.The natural language recognition module 146 parses a text input anddetermines whether the input contains a specified text string. Forexample, the natural language recognition module 146 may be used todetermine if a state of the UI includes a specific text string.

The headless browser 126 is a web browser that does not have a graphicaluser interface. The headless browser 126 may include a command-lineinterface used to load and interact with web pages, such as web pages ofa web application hosted on a web server 158.

The test creation module 128 obtains test parameters and generates testconfiguration data. The test parameters may include a starting addressof the UI, a goal, a maximum number of permitted steps to the goal, thenumber of tests to be conducted, and variable data. The variable datamay include text strings—such as usernames or passwords—or menu optionsthat may be entered into data entry fields of the UI. In someimplementations, the test creation module 128 may receive the testparameters from the operator device 114. In other implementations, thetest creation module 128 may retrieve previously stored test parametersfrom the storage device 112. The test creation module 128 provides thetest configuration data to the test execution module 130. The testcreation module 128 may also store the test configuration data in thestorage device 112.

The test execution module 130 uses the test configuration data generatedby the test creation module 128 to perform tests on the UI to determinea shortest path in the UI to the specified goal. In response toreceiving the test configuration data from the test creation module 128,the test execution module 130 initiates the first test of the UI. Forexample, the test execution module 130 instructs the headless browser126 to load the starting page of the UI. The headless browser 126 loadsthe requested web page and provides a document object model (DOM) of theloaded web page to the test execution module 130.

The test execution module 130 sets a position of the cursor to apredetermined position (such as the center) of the UI, marks eachvariable in the test configuration as unused, and sets the state of theneural networks in the neural network module 144 to untrained. The testexecution module 130 uses the analysis module 132 to analyze the currentstate of the UI—in other words, the currently loaded web page. Theanalysis module 132 may use the DOM received from the headless browser126 to perform the analysis.

The analysis module 132 first determines whether the current state ofthe UI satisfies the goal criteria—for example, that the loaded web pageincludes goal text. The analysis module 132 may use the natural languagerecognition module 146 to parse the text of the loaded web page todetermine whether it includes the goal text. In response to locating thegoal text, the analysis module 132 determines that the current test is asuccess. The current test is considered a failure when a maximum numberof steps has been reached without the goal text being found. In responseto determining that the current test is neither a success or failure,the analysis module 132 determines whether the current state of the UIincludes interactive elements—for example, links, data entry fields, andelements that submit data—that can be executed to change the state ofthe UI.

The analysis module 132 assigns a probability to an action associatedwith each interactive element located in the loaded page. Theprobability represents the likelihood that executing the action willresult in reaching the goal. If the neural networks in the neuralnetwork module 144 are trained, the analysis module 132 uses the neuralnetworks in the neural network module 144 to determine the probability.The analysis module 132 also assigns a weighting to each type of actionlink, entry field, and submit. If the neural networks in the neuralnetwork module 144 are trained, the analysis module 132 uses the neuralnetworks in the neural network module 144 to determine the weight foreach action type. The analysis module 132 then determines a weightedprobability for each action. The analysis module 132 provides eachidentified action and the associated weighted probability to the testexecution module 130. If the neural networks in the neural networkmodule 144 are not yet trained, the analysis module 132 uses the MonteCarlo module 142 to assign random values for the probabilities of actionsuccess and action type success.

The test execution module 130 selects the action with the highestprobability among the actions provided by the analysis module 132. Thetest execution module 130 determines the distance between the currentposition and the nearest point of the interactive element associatedwith the selected action. The test execution module 130 updates a totaldistance of the current test based on the calculated distance and thenstores both the calculated distance as well as the total distance. Thetest execution module 130 instructs the headless browser to perform theselected action. The executed action is referred to as a step and thescreen of the UI that results from the execution of the action is astate. In other words, performing a step results in transitioning the UIfrom a first state to a second state or results in transforming thefirst state.

The test execution module 130 concludes that the current test is afailure (that is, not a shortest pat) if the total distance of thecurrent test is greater than a previously identified path to the goal.Otherwise, the test execution module 130 proceeds with the test andanalyzes the current state of the UI, as described above.

In response to determining that the current test is either a success orfailure, the test execution module 130 logs the results of the completedtest. In response to determining that the current test was a success,the test execution module 130 stores the steps of the current test asthe shortest path to the goal. The test execution module 130 thendetermines whether the number of completed tests has reached the maximumnumber of tests. If so, the test execution module 130 outputs theshortest path found to the goal.

If more tests are needed, the test execution module 130 begins a newtest. If the just-completed test was a success, the training module 134trains (or, after the first training, retrains or supplements thetraining of) the neural networks in the neural network module 144. Thetraining module uses the results of the completed, successful test togenerate training data. The training module 134 saves the neuralnetworks in the neural network module 144 and the associated state ofthe neural networks—such as trained or untrained. In someimplementations, the training module 134 may store both the neuralnetworks and their associated state in storage device 112.

The training module 134 then retrains the link neural network 148, thesubmit neural network 150, the data match neural network 152, the datachoice neural network 154, and the action weight neural network 156using the generated training data. Once the neural networks have beenretrained, the training module 134 determines the performance of theretrained networks. For example, the training module 134 sets the stateof the neural networks in the neural network module 144 to trained. Thetraining module 134 then uses the analysis module 132 to analyze eachstep of each completed test and produces a list of possible actions andassociated probabilities. Since the status of the neural networks in theneural network module 144 is set to trained, the probabilities producedby the analysis module 132 will be based on the newly trained networks.For each action in each analyzed step, the training module 134calculates a change between the previously determined probability andthe newly determined probability.

The training module 134 determines the performance of the networks basedon the calculated changes. In response to determining that retaining didnot improve the performance of the neural networks in the neural networkmodule 144, the training module restores the neural networks in theneural network module 144 to their previous state. For example, thetraining module may retrieve the stored networks and associated statefrom the storage device 112.

Next, the reweighting module 136 recalculates probabilities, such as forevery action in every state of the last completed test. The reweightingmodule 136 calculates the probabilities using the neural networks in theneural network module 144, if they are trained, or updates theprobabilities with new random values. The use of the neural networks inthe neural network module 144 will result in probabilities that moreclosely match the actual probability that each action will result in thegoal.

The test execution module 130 selects an action among the reweightedactions. The test execution module 130 bases the selection on thereweighted probability of each action and the total distance required toreach the action from the initial state of the UI.

The session creation module 138 determines the steps required to reachthe action selected by the test execution module 130 from the initialURL of the UI. The session creation module 138 executes the determinedsteps required to reach the action and the headless browser 126 returnsthe DOM of the resulting web page. The analysis module 132 then analyzesthe current state of the UI, as described above. The test executionmodule 130 continues to execute new tests until the maximum number oftests has been reached.

The UI enhancement module 122 generates prompts and other aids that canbe used to guide a user to a goal in a UI based on a shortest path tothe goal and eye tracking experiments that correspond to the UI. Eyetracking experiments are a form of UI usability tests that includetracking data based on either the gaze of a user or the movement of amouse cursor by the user while the user is navigating a UI to reach agoal. The eye tracking experiments are used to identify where a user ofthe UI is likely to deviate from the shortest path to the goal. The UIenhancement module 122 determines a distance for each page of the UIthat represents a deviation from the shortest path that may requirereinforcement of a shortest path step to aid the user in reaching thegoal.

The UI enhancement module 122 includes a reinforcement module 160 and aprompt generation module 162. The reinforcement module 160 determineswhich steps of a shortest path may need reinforcement. The reinforcementmodule 160 bases the determination on the eye tracking experimentsassociated with the UI. The UI enhancement module 122 determines whichsteps of the shortest path require reinforcement based on thedetermination of the reinforcement module 160 and a receivedreinforcement percentage.

The reinforcement module 160 includes a distance module 164 and a stepanalysis module 166. The distance module 164 determines a reinforcementdistance for each URL of the UI based on the eye tracking experimentsassociated with the goal and the shortest path to the goal. Thereinforcement distance is a distance from a shortest path step—forexample, the distance between a cursor and a point on the shortestpath—that indicates that the step may require reinforcement.

The distance module 164 includes a prediction module 168 and a clusteranalysis module 170. The distance module 164 obtains parameters fordetermining the distance, such as weighting values for a fitnessalgorithm and a permitted number of evolutions. An evolution consists ofthe selection of random reinforcement distances for each URL of the UIthat are then used by the prediction module 168 to predict the successor failure of a single eye tracking experiment. The number of permittedevolutions represents the number of different sets of randomreinforcement distances that are generated and used to predict theoutcome of different eye tracking experiments.

The distance module 164 compares tracking data in each eye trackingexperiment to the shortest path and determines a distance between eachsample included in the eye tracking data and a corresponding point onthe shortest path. The distance module 164, for each URL of the UI,calculates the median distance between eye tracking samples associatedwith successful tests and the shortest path. Successful tests includetests in which the user reached the goal in the UI—for example, the usersuccessfully logged in to a web application. The distance module 164,for each URL of the UI, also calculates the median distance between eyetracking samples associated with failed tests and the shortest path.Failed tests include tests in which the user did not reach the goal—forexample, the user was unable to log in to the web application. Failedtests may also include tests in which the user reached the goal, but didnot follow the shortest path to the goal. For example, although the usermay have eventually logged in to the web application, they did not do soin the fewest number of steps possible.

The distance module 164 uses the determined median distances for eachURL as upper and lower bounds for generating random reinforcementdistances for each URL. For example, the distance module 164 set thelower bound for a URL equal to the median success distance for the URLand set the upper bound for the URL equal to the median failuredistance. In some implementations, the distance module 164 may use theMonte Carlo module 142 to generate the random distances.

The prediction module 168 then performs an evolution using the randomlygenerated reinforcement distances and the first eye tracking experiment.For each sample in the eye tracking experiment, the prediction module168 determines whether the randomly generated reinforcement distancesand the shortest path indicates that the experiment was a success (theuser reached the goal) or a failure (the user either did not reach thegoal or did not follow the shortest path to the goal). The predictionmodule 168 then compares the predicted outcome of the experiment withthe stored outcome of the experiment. For each evolution, the predictionmodule 168 outputs the number of times that the correct outcome of theexperiment was predicted, the number of times the experiment wasincorrectly predicted to be a success, and the number of times theexperiment was incorrectly predicted to be a failure. The reinforcementmodule 160 stores the output of the prediction module 168 after eachevolution.

The distance module 164 continues generating random reinforcementdistances for each URL and the prediction module 168 performs anevolution using the randomly generated reinforcement distances and a neweye tracking experiment until the maximum number of permitted evolutionshas been reached.

In response to performing the maximum number of permitted evolutions,the distance module 164 determines a fitness value for each randomlygenerated reinforcement distance using the stored outputs of theprediction module 168 and the obtained weighting values. The clusteranalysis module 170 then performs a cluster analysis of a plot of eachfitness value versus the test distance associated with the fitnessvalue. In some implementations, the cluster analysis module 170 may usethe density-based spatial clustering of applications with noise (DBSCAN)algorithm to perform the cluster analysis. In other implementations, thecluster analysis module 170 may use the mean shift technique to performthe cluster analysis. In yet other implementations, the cluster analysismodule 170 may use another suitable cluster analysis technique oralgorithm to analyze the plot of fitness values versus randomlygenerated distances.

In response to determining that the cluster analysis module 170identified a single cluster of fitness values versus reinforcementdistances for each URL of the UI, the distance module 164 outputs areinforcement distance for each URL. For each URL, the distance module164 outputs the reinforcement distance associated with the URL with thehighest fitness value.

In response to determining that the cluster analysis module 170 did notidentify a single cluster of fitness values versus reinforcementdistances for each URL, the distance module 164 sets new upper and lowerbounds for each URL. For each URL, the distance module 164 sets theupper bound to the largest distance in the most successful clusterassociated with the URL and sets the lower bound to the smallestdistance in the most successful cluster. The most successful cluster isthe cluster that includes the largest number of data points.

The distance module 164 resets the number of performed evolutions tozero. The distance module 164 then begins generating randomreinforcement distances for each URL using the new upper and lowerbounds and the prediction module 168 performs evolutions using therandomly generated reinforcement distances until the maximum number ofpermitted evolutions has been reached.

The distance module 164 determines fitness values for the newlygenerated random reinforcement distances based on the output of theprediction module 168. The cluster analysis module 170 then analyzes thenew fitness values. The process continues in this manner until thedistance module 164 outputs a reinforcement distance for each URL.

The step analysis module 166 determines, based on the reinforcementdistance for each URL, the number of times that the eye trackingexperiments indicate that a user deviated from a step of the shortestpath. For example, the step analysis module 166 compares each sampleincluded in an eye tracking experiment to an associated step in theshortest path and determines, based on the reinforcement distance forthe URL associated with the shortest path step, whether the sampleindicates that the user deviated from the shortest path step.

The prompt generation module 162 generates prompts for each step of theshortest path. The prompts may be used to aide a user of the UI incompleting the step. The prompt generation module 162 generates theprompts based on the action included in the step of the shortest path.For example, the prompt generation module 162 may generate a promptinstructing a user to click on a link included in a shortest path step.As another example, the prompt generation module 162 may instruct theuser to enter text into a data entry field—for example, typing in ausername.

The prompt generation module 162 may use a DOM element that correspondsto the action included in the shortest path step to generate the prompt.In some implementations, the prompt generation module 162 includesdetails of the interactive element of the UI obtained from the DOMelement—for example, the name of or text associated with the element—inthe generated prompt. The prompt generation module 162 may use theheadless browser 126 to obtain the DOM element.

The UI enhancement module 122 determines if the step requiresreinforcement based on the reinforcement percentage and the results ofthe step analysis module 166. For example, the UI enhancement module 122calculates a ratio of the number of times an eye tracking sampleindicated that a user deviated from the shortest path step and the totalnumber of eye tracking samples associated with the step. The UIenhancement module 122 determines that the step requires reinforcementin response to determining that the calculated ratio is equal to orgreater than the reinforcement percentage.

In response to determining that a step of the shortest path requiresreinforcement, the UI enhancement module 122 generates a reinforcementfor the step. In some implementations, the reinforcement may include anadditional prompt that includes more information about the action to beexecuted by the user. For example, the additional prompt may inform theuser of the location of the interactive element on the screen. In otherimplementations, the reinforcement may include instructions to highlightthe interactive element. In yet other implementations, the reinforcementmay include both the additional prompt and the highlightinginstructions. The UI enhancement module 122 stores the generated promptsand reinforcements, along with the shortest path. For example, the UIenhancement module 122 may store the information in the storage device112.

In various implementations, the shortest path generated by the shortestpath module 120 and the prompts and reinforcements generated by the UIenhancement module 122 may be used to alter the UI to improve theusability of the UI and the associated application. For example,interactive elements of the UI—such as links, buttons, and data entryfields—may be repositioned, either automatically or manually, to reducethe total distance required to reach a goal. The UI testing andenhancement device 110 may then perform additional tests on the alteredUI. For example, the shortest path module 120 may generate a newshortest path for the altered UI to determine if the repositionedelements improved the UI—in other words, whether total distance of theshortest path was reduced.

Additionally, new eye tracking experiments may be performed on a UI thatincorporates the generated UI enhancements. The UI enhancement module122 may use the data from the new eye tracking experiments to determineif the enhancements improved the usability of the UI. For example, theUI enhancement module 122 may determine that UI enhancements resulted inan overall reduction in the deviation from the shortest path—that is,fewer steps of the shortest path require reinforcement.

Neural Networks

In various implementations, the link neural network 148, the submitneural network 150, the data choice neural network 154, and the actionweight neural network 156 may be implemented as long short-term memory(LSTM) neural networks. LSTM neural networks are feedforward neuralnetworks. FIG. 2 is a functional block diagram of a generic example LSTMneural network 202. The LSTM neural network 202 includes an input layer204, a hidden layer 208, and an output layer 212. The input layer 204includes inputs 204 a, 204 b . . . 204 n. The hidden layer 208 includes“neurons” 208 a, 208 b . . . 208 n. The output layer 212 includesoutputs 212 a, 212 b . . . 212 n.

Each neuron of the hidden layer 208 receives an input from the inputlayer 204 and outputs a value to the corresponding output in the outputlayer 212. For example, the neuron 208 a receives an input from theinput 204 a and outputs a value to the output 212 a. Each neuron, otherthan the first neuron 208 a, also receives the output of the previousneuron as an input. For example, the neuron 208 b receives an input fromthe input 204 b and an input from the output 212 a. In this way theoutput of each neuron is fed forward to the next neuron in the hiddenlayer 208. The last output 212 n in the output layer 212 outputs aprobability associated with the inputs 204 a-204 n. Although the inputlayer 204, the hidden layer 208, and the output layer 212 are depictedas each including three elements, the input layer 204, the hidden layer208, and the output layer 212 may contain any number of elements. Invarious implementations, each layer of the LSTM neural network 202 mustinclude the same number of elements as each of the other layers of theLSTM neural network 202.

FIG. 3 is a functional block diagram of an example implementation of thelink neural network 148. The link neural network 148 includes an inputlayer 304, a hidden layer 308, and an output layer 312. The input layer304 includes inputs 304 a, 304 b, and 304 c. The hidden layer 308includes neurons 308 a, 308 b, and 308 c. The output layer 312 includesoutputs 312 a, 312 b, and 312 c.

The link neural network 148 is depicted as processing the link“company.com/order/display.htm?number=123456”. The link is parsed tocreate inputs for inputs 304 a, 304 b, and 304 c. For example, thestatic portion of the link—“company.com”—is discarded. The remainingportion of the link is separated into inputs based on the format andsyntax of the link. Specifically, “order” is provided to input 304 a,“display.htm” is provided to input 304 b, and “number=123456” isprovided to input 304 c.

In the example depicted in FIG. 3, the output of the link neural network148 is 0.75, which represents the probability that execution of theinputted link will result in the goal. Although the link neural network148 is depicted as including three neurons, the link neural network 148may include more or fewer neurons.

FIG. 4 is a functional block diagram of an example implementation of thesubmit neural network 150. The submit neural network 150 includes aninput layer 404, a hidden layer 408, and an output layer 412. The inputlayer 404 includes inputs 404 a, 404 b, and 404 c. The hidden layer 408includes neurons 408 a, 408 b, and 408 c. The output layer 412 includesoutputs 412 a, 412 b, and 412 c.

The submit neural network 150 is depicted as processing a submit namedlogin located at “company.com/help/faq.htm”. The name of the submit isprovided to input 404 a. The location of the submit is parsed to createinputs for inputs 404 b and 404 c. For example, the static portion ofthe location—“company.com”— is discarded and the remaining portion ofthe location is separated into different inputs based on the format andsyntax of the location. Specifically, “help” is provided to input 404 band “faq.htm” is provided to input 404 c.

In the example depicted in FIG. 4, the output of the submit neuralnetwork 150 is 0.25, which represents the probability that execution ofthe inputted submit will result in the goal. Although the submit neuralnetwork 150 is depicted as including three neurons, the submit neuralnetwork 150 may include more or fewer neurons.

FIG. 5A is a functional block diagram of an example implementations ofthe data choice neural network 154. The data choice neural network 154includes an input layer 504, a hidden layer 508, and an output layer512. The input layer 504 includes inputs 504 a, 504 b, and 504 c. Thehidden layer 508 includes neurons 508 a, 508 b, and 508 c. The outputlayer 512 includes outputs 512 a, 512 b, and 512 c.

The data choice neural network 154 is depicted as processing a text dataentry field named “username” located at “company.com/help/faq.htm”. Thename of the data entry field is provided to the input 504 a. The dataentry field type—text—is provided to the input 504 b. The location ofthe data entry is parsed to create an input for the input 504 c. Forexample, the static portion of the location—company.com—is discarded andthe remaining portion of the location is separated, if necessary, intoinputs based on the format and syntax of the location. In the exampledepicted in FIG. 5A, the remaining portion of the location does notrequire separation. In other words, “home.htm” is simply provided to theinput 504 c.

In the example depicted in FIG. 5A, the output of the data choice neuralnetwork 154 is 0.98, which represents the probability that entering textinto the inputted data entry field will result in the goal. Although thedata choice neural network 154 is depicted as including three neurons,the data choice neural network 154 may include more than three neurons.

FIG. 5B is a functional block diagram of an example implementation ofthe data match neural network 152. The data match neural network 152 isa convolutional neural network. Similar to LSTM neural networks,convolutional neural networks include an input layer, a hidden layer,and an output layer. However, in a convolutional neural network, theoutput layer includes one fewer output than the number of neurons in thehidden layer and each neuron is connected to each output. Additionally,each input in the input layer is connected to every neuron in the hiddenlayer.

The data match neural network 152 includes an input layer 524, a hiddenlayer 528, and an output layer 532. In FIG. 5B, the input layer 524includes inputs 524 a, 524 b, and 524 c; the hidden layer 528 includesneurons 528 a, 528 b, and 528 c; and the output layer 532 includesoutputs 532 a and 532 b.

The data match neural network 152 is depicted as processing a text dataentry named “username” that is paired with the data “testuser.” Thepaired data—“testuser”—is provided to the input 524 a. The data entryfield name—“username”—is provided to the input 524 b. The data entryfield type—test—is provided to the input 524 c. In the example depictedin FIG. 5B, the output of the data match neural network 152 is 0.75,which represents the probability that the inputted data should beentered into the inputted data entry field.

FIG. 6 is a functional block diagram of an example implementation of theaction weight neural network 156. The action weight neural network 156includes an input layer 604, a hidden layer 608, and an output layer612. The input layer 604 includes inputs 604 a, 604 b, and 604 c. Thehidden layer 608 includes neurons 608 a, 608 b, and 608 c. The outputlayer 612 includes outputs 612 a, 612 b, and 612 c.

The action weight neural network 156 is depicted as processing a submitaction type that is located at “company.com/help.faq.htm”. The actiontype is provided to the input 604 a. The location of the action isparsed to create inputs for the inputs 604 b and 604 c. For example, thestatic portion of the location—“company.com”—is discarded and theremaining portion of the location is separated into inputs based on theformat and syntax of the location. Specifically, “help” is provided tothe input 604 b and “faq.htm” is provided to the input 604 c.

In the example depicted in FIG. 6, the output of the action weightneural network 156 is 0.25, which represents the probability thatexecuting a submit located at “company.com/help/faq.htm” will result inthe goal. Although the action weight neural network 156 is depicted asincluding three neurons, the action weight neural network 156 mayinclude more than three neurons and as few as two neurons.

Shortest Path Determination

FIGS. 7A and 7B are a flowchart depicting an example method ofdetermining a shortest path to a goal in a UI. Although the examplemethod is described below with respect to the UI testing and enhancementdevice 110, the method may be implemented in other devices and/orsystems. In various implementations, control may be performed by theshortest path module 120.

Control begins at 703 of FIG. 7A. At 703, control obtains testparameters: for example, goal text, data for a set of variables, aninitial URL of a UI to be tested, the maximum number of permitted stepsper test, and the maximum number of permitted tests. Control then storesthe test parameters as test configuration data. For example, control maystore the test configuration data in the storage device 112. Controlcontinues with 706, where control initializes values associated with thetests. For example, control sets the training status of the neuralnetworks (Networks_Trained) to false and Completed_Tests to Null.Control then continues with 709, where control loads the initial URL andsets the number of executed tests (Tests_Ran) to zero.

At 709, control, for each variable in the stored configuration data,sets a used status flag (Used) to false. Control then continues with712, where control sets the current position (Current_Position) to thecenter of the UI. At 712, control also stores the result of loading theinitial URL, the current state of the UI, as Initial_State, and thencontrol continues with 715. At 715, control initializes the dataassociated with the first test: for example, control sets Test_Data toNull, Step to zero, and Test_Distance to zero. Control then progressesto 718.

At 718, control initializes the data associated with the currentstep—for example, control sets Step_Data to Null. Control then analyzesthe current state of the UI. For example, control may perform the methoddisclosed in FIGS. 8A-8C. The outcome of the analysis may be Success,Failure, or a list of actions. Success indicates that the current stateof the UI includes the goal text or satisfies some other goal criteria.Failure may indicate that the current state includes neither the goaltext nor any possible actions—such as traversing links, entering datainto a data entry field, or submitting data that has been entered into adata entry field. Failure may instead indicate that the test hasperformed the max number of permitted steps and has not reached thegoal.

A list of actions indicates that the current state includes at least onepossible action that may be executed. Each action in the list of actionsincludes a probability that the action will result in reaching the goal.Control stores the results of the analysis. For example, control addsthe outcome of success or failure to Test_Data and an outcome of a listof actions to Step_Data. Control then progresses to 721, where controldetermines whether the outcome of the analysis is a list of actions or aSuccess/Failure indicator. If the outcome is a list of actions, controlprogresses to 724 of FIG. 7B; otherwise, control transfers to 727.

At 724 of FIG. 7B, control determines an action to execute. For example,control selects the action with the highest overall probability amongthe listed actions. Control may use equations 1 and 2 below to determinethe overall probability for each action.

$\begin{matrix}{P_{{distance}\_{weighted}} = \left\{ \begin{matrix}P_{{action},} & {{Test\_ Distance} = 0} \\{{P_{action}*\left( \frac{Test\_ Distance}{Action\_ Distance} \right)},} & {{Test\_ Distance} > 0}\end{matrix} \right.} & (1) \\{\mspace{79mu}{P_{overall} = \frac{P_{weighted}}{\sum P_{{distance}\_{weighted}}}}} & (2)\end{matrix}$

P_(action) is the probability of an action included in the list ofactions. Action_Distance is the Euclidean distance between the currentposition and the nearest point of the UI element associated with theaction. Test_Distance is the total distance traveled in the UI duringthe current test. ΣP_(distance_weighted) is a summation ofP_(distance_weighted) for each of the actions in the list of actions.

Control then progresses to 730 where control determines whether theselected action is a submit, a link, or an entry. In response todetermining that the selected action is a submit, control progresses to733; in response to determining that the selected action is a link,control progresses to 745; otherwise, control progresses to 748.

At 733, control determines a point of the UI element associated with thesubmit that is nearest to the current position (Current_Position). Invarious implementations, the coordinates of the nearest point areobtained from the list of actions for the selected action. Control thencalculates the step distance (Step_Distance) as the Euclidean distancebetween Current_Position and the nearest point. Control then continueswith 739, where control stores the step distance and executes thesubmit—for example, control adds Step_Distance to Step_Data and triggersa click on the UI element associated with the submit. At 739, controlalso updates Current_Position to the coordinates of the nearest point.

Returning to 736, control determines the point of the UI elementassociated with the link that is nearest to Current_Position. Thecoordinates may be stored in the list of actions for the selectedaction. Control then sets the step distance (Step_Distance) to theEuclidean distance between Current_Position and the nearest point.Control then continues with 740, where control stores the step distanceand executes the link—for example, adds Step_Distance to Step_Data andtriggers a click on the UI element associated with the link. At 740,control also sets Current_Position to the nearest point of the UIelement associated with the link. For example, control may setCurrent_Position to the coordinates stored in the list of actions forthe selected action. Control then progresses to 742.

Returning to 738, control determines the point of the UI elementassociated with the entry that is nearest to Current_Position. Forexample, the coordinates stored in the list of actions for the selectedaction. Control then sets the step distance (Step_Distance) to theEuclidean distance between Current_Position and the nearest point of theUI element associated with the entry. Control then continues with 741where control stores the step distance and executes the entry—forexample, adds Step_Distance to Step_Data and enters the variableassociated with the action into the data entry field. At 741, controlalso marks the variable entered into the data entry field as used andsets Current_Position to the nearest point of the UI element associatedwith the entry. For example, control may set Used for the variable toTrue and Current_Position to the coordinates stored in the list ofactions for the selected action. Control then progresses to 742.

At 742, control stores the progress of the current test—for example,control adds Step_Data to Test_Data, adds Step_Distance toTest_Distance, and increases the number of steps taken by 1. Controlthen continues with 756 of FIG. 7A.

At 756 of FIG. 7A, control determines whether the current test distanceis longer than an already determined path—in other words, whether thedistance of the current test is longer than the shortest completedsuccessful test. If so, control progresses to 757; otherwise controlreturns to 718. At 757, control sets the status of the current test to afailure. For example, control sets Outcome equal to failure. Controlthen continues with 727.

Returning to 727, control stores the current test as a completed test.For example, control adds the total distance of the test (Test_Distance)and the data associated with the current test (Test_Data) toCompleted_Tests. At 727, control increments Tests_Ran by one. Controlprogresses to 758 where control determines whether the number of teststhat have been run (Tests_Ran) is equal to or greater than the number ofpermitted tests (Max_Tests). If so, control progresses to 761; otherwisecontrol transfers to 764.

At 761, control determines a shortest path to the goal based on thestored completed tests (Completed_Tests). For example, control selects asuccessful test stored in Completed_Tests with the smallest totaldistance (Test_Distance) as the shortest path. Control progresses to762, where control outputs the stored data associated with the selectedtest (Test_Data) as the shortest path. Control then ends.

At 764, control trains the neural networks. For example, controlperforms the method disclosed in FIGS. 9A and 9B. Control thenprogresses to 770 where control reweights the probabilities associatedwith the actions of every step of the completed test—for example,control may perform the method disclosed in FIGS. 10A-10C. Control thencontinues with 773, where control selects the action associated with thecompleted test that has the highest overall probability of resulting inthe goal text. Control may use equations 1 and 2, as previouslydescribed, to determine the overall probability of each action. Controlthen progresses to 776, where control builds a new session based on theselected action. For example, control may perform the method disclosedin FIG. 11. Control then returns to 715.

FIGS. 8A-8C are a flowchart depicting an example method of analyzing astate of a UI. Although the example method is described below withrespect to the UI testing and enhancement device 110, the method may beimplemented in other devices and/or systems. In various implementations,control may be performed by the shortest path module 120 and/or theanalysis module 132.

Control begins at 802 of FIG. 8A. At 802, control initializes the actiondata for the state to be analyzed. For example, control sets Action_Datato Null. Control then progresses to 804, where control searches thecurrent state—the screen—of the UI to determine whether the goalcriteria have been satisfied. For example, in FIG. 8A, controldetermines whether the current state includes the goal text. Control mayuse the natural language recognition module 146 to match text found onthe screen to the goal text. If, at 806, the goal text was found,control continues with 808; otherwise, control transfers to 810.

At 808, control updates the distance of the current test based on thelocation of the goal text. For example, control calculates the Euclideandistance between Current_Position and the point of the goal text that isnearest to Current_Position. Control adds the distance to Test_Distance.At 808, control also sets Outcome to Success to indicate that thecurrent state includes the goal text. Control progresses to 809 wherecontrol outputs the value of Outcome. Control then ends.

Returning to 810, control determines whether the maximum number ofpermitted steps (Max_Steps) has been reached. If the number of steps inthe current session is less than the maximum number of steps, controlprogresses to 816; otherwise, control transfers to 814. At 814, controlsets Outcome to Failure. Control then progresses to 809.

At 816, control searches for links in the current state and then controlprogresses to 818. At 818, if one or more links were located in thecurrent state, control continues with 820; otherwise, control transfersto 822 of FIG. 8B.

At 820, control generates a list of links found in the current state.Control removes any link from the list that has been previously visitedduring the current session. Control then continues with 824 wherecontrol, for each link in the list, determines and stores thecoordinates of a point of an associated link UI element that is nearestto Current_Position. Control then progresses to 826, where controldetermines whether the neural networks are trained. If so, controlprogresses to 828; otherwise, control transfers to 830. At 828, controluses the link neural network 148 to estimate for each link in the list,the probability that the link will result in the goal. Control storesthe probability for each link in the link list. Control then continueswith 832.

At 830, control, for each link in the link list, assigns a randomprobability that the link will result in the goal and stores theprobability in the link list. In some implementations, control may usethe Monte Carlo module 142 to assign the random probabilities. Controlcontinues with 832. At 832, control stores the link list—for example,control adds the link list and associated probabilities to Action_Data.Control progresses to 822 of FIG. 8B.

At 822 of FIG. 8B, control searches for data entry fields in the currentstate and then control progresses to 834. At 834, control determineswhether any data entry fields were located in the current state. If so,control continues with 836; otherwise, control transfers to 838 of FIG.8C.

At 836, control determines if the configuration data includes data thatmay be entered into the located data entry field—that is, variables notalready marked as used. If so, control progresses to 840; otherwise,control transfers to 838 of FIG. 8C. At 840, control generates a list ofall possible data entry fields and available data combinations. For eachentry in the list, control determines and stores the coordinates of thenearest point of the data entry field to Current_Position. Controlcontinues with 842.

At 842, control determines whether the neural networks are trained. Ifso, control progresses to 844; otherwise, control transfers to 846. At844, for each data entry field and data combination in the list, controluses the data match neural network 152 to estimate the probability thatthe data entry field matches the data (P_match). Control progresses to848 where, for each data entry field in the list, control uses the datachoice neural network 154 to estimate the probability that entering datainto the data entry field will result in the goal (P_choice). Controlprogresses to 850.

At 846, for each entry field data combination in the list, controlassigns a random probability that the entry field matches the data(P_match). Control progresses to 852 where, for each entry field in thelist, control assigns a random probability that entering data into theentry field will result in the goal text (P_choice). In someimplementations, control may use the Monte Carlo module 142 to generaterandom probabilities for P_match and P_choice. Control continues with850.

At 850, control calculates and stores a combined probability for eachdata entry field and data combination in the list. For example, controlmay average P_match and P_choice for each item in the list. Control thenprogresses to 854 where control stores the data entry field and datacombination list—for example, control may add the list to Action_Data.Control continues with 838 of FIG. 8C.

At 838 of FIG. 8C, control searches for “submits”—for example, buttonsor menu options that submit data entered into a data entry field—in thecurrent state and then control progresses to 856. At 856, controldetermines whether any submits were located in the current state. If so,control continues with 858; otherwise, control transfers to 860.

At 858, control generates a list of submits found in the current state.Control continues with 862 where control, for each submit in the list,determines and stores the coordinates of the nearest point of the submitUI element to Current_Position. Control progresses to 864, where controldetermines whether the neural networks in the neural network module 144are trained. If so, control progresses to 866; otherwise, controltransfers to 868. At 866, control uses the submit neural network 150 toestimate the probability that each submit in the list will result in thegoal. Control stores the probabilities in the submit list. Controlcontinues with 870.

At 868, control assigns a random probability to each submit in the listand stores the probability in the submit list. In some implementations,control may use the Monte Carlo module 142 to assign the randomprobabilities. Control continues with 870.

At 860, control determines whether Action_Data is equal to Null—in otherwords, the current state fails to contain any links, data entry fieldand data combinations, or submits. If so, control progresses to 874;otherwise, control transfers to 872. At 874, control sets Outcome toFailure and then outputs Outcome. Control then ends.

Returning to 870, control stores the submit list—for example, controlmay add the submit list to Action_Data. Control continues with 872. At872, control determines whether the neural networks in the neuralnetwork module 144 are trained. If so, control continues with 876;otherwise control transfers to 878. At 876, control uses the actionweight neural network 156 to determine a probability weighting for eachaction type—for example, links, data entry fields, and submits. Controlthen progresses to 880.

At 878, control assigns a random probability weighting to each actiontype—for example, links, entry fields, and submits. In someimplementations, control may use the Monte Carlo module 142 to determinethe random probability weightings. Control then progresses to 880.

At 880, control determines and stores a weighted probability for eachstored action—in other words, for each link, data entry field and datacombination, and submit stored in Action_Data. For example, control mayuse equations 3 and 4 below to calculate each weighted probability(P_(action)).

$\begin{matrix}{P_{weighted} = {W_{type}*P_{unweighted}*\left( \frac{\sum({Distance})}{Distance} \right)}} & (3) \\{P_{action} = \frac{P_{weighted}}{\sum P_{weighted}}} & (4)\end{matrix}$

P_(unweighted) is the stored probability for the action, W_(type) is theprobability weighting for the action type of the action, and Distance isthe Euclidean distance between the current position (Current_Position)and the nearest point stored for each action in Action_Data. Equation 4is used to normalize the weighted probability (P_(weighted)) of eachaction.

Control continues with 882 where control outputs the results of theanalysis of the current state of the UI. For example, control setsOutcome to Action_Data and then outputs Outcome. Control then ends.

FIGS. 9A and 9B are a flowchart depicting an example method of trainingthe neural networks in the neural network module 144 based on completedshortest path tests. Although the example method is described below withrespect to the UI testing and enhancement device 110, the method may beimplemented in other systems. In various implementations, control may beperformed by the shortest path module 120 and/or the training module134.

Control begins at 904 of FIG. 9A in response to control receiving acompleted test. At 904, control determines whether the completed testwas a success. If so, control continues with 908; otherwise, controlends. In other words, only successful tests are used to train neuralnetworks in the neural network module 144. At 908, control sets State tothe number of states in the completed test—for example, the number ofstates stored in Test_Data. Control also sets current state(Session_State) to the first state of the successful test—for example,the first step stored in Test_Data.

Control then continues with 912, where control determines the actiontype associated with an action executed in the current state(Session_State). Control adds the action weight of the determined actiontype to training data. Control then progresses to 916 where controlsaves the unweighted probability of the executed action and any dataassociated with the executed action—for example, a link, a data entryfield and data combination, or a submit—to the training data. Controlcontinues with 920, where control decrements State by one. Control thenprogresses to 924.

At 924, control determines whether there are any additional states inthe completed test. For example, control determines whether State isgreater than zero. If so, control progresses to 928; otherwise, controltransfers to 930. At 928, control sets Session_State to the next stateof the completed test—for example, the next state stored in Test_Data.Control then returns to 912.

At 930, control stores the current training status of the neuralnetworks in the neural network module 144. Control progresses to 932,where control determines whether the neural networks in the neuralnetwork module 144 are trained—that is, whether the value ofNetworks_Trained is true. If so, control progresses to 936; otherwise,control transfers to 940. At 936, control saves the current neuralnetworks. Control then progresses to 940.

At 940, control uses the training data to train the neural networks.Control progresses to 944, where control sets Test to the number ofcompleted tests—for example, the number of tests stored inCompleted_Tests. Control uses Training_Test to step through each storedtest. At 944, control also sets Training_Test to the first test storedin Completed_Tests. Control continues with 948 of FIG. 9B.

At 948 of FIG. 9B, control sets the status of the neural networks in theneural network module 144 to trained. At 948, control also setsChange_Success to zero and Change_Failure to zero. Change_Success andChange_Failure are used to store the differences between the previouslycalculated probabilities and the probabilities that the newly trainedneural networks produce. Control continues with 952 where control setsTraining_Session to the Test_Data associated with Training_Test. At 952,control also sets State to the number of states stored inTraining_Session. Control uses State to step through each state storedin Training_Session. Control progresses to 956 where control setsTraining_State to the first state stored in Training_Session. Controlcontinues with 960.

At 960, control executes the action associated with Training_State andcontrol continues with 964. At 964, control analyzes the state of the UIafter the action is executed—for example, control may perform the methoddisclosed in FIGS. 8A-8C. Control then progresses to 966 where controldetermines whether the test associated with the action(Training_Session) was a success. If so, control continues with 968;otherwise, control transfers to 970. At 968, for each action identifiedduring the analysis, control calculates the difference between theprobability stored in Training_State and the probability generated bythe analysis. Control then performs a summation of all of thedifferences and adds the value of the summation to Change_Success.Control progresses to 972.

At 970, for each action identified during the analysis, controlcalculates the difference between the probability generated by theanalysis and the probability stored in Training_State. Control thenperforms a summation of the differences and adds the value of thesummation to Change_Failure. Control then continues with 972.

At 972, control decrements State by 1. Control continues with 976 wherecontrol determines whether there are any additional states stored inTraining_Session—in other words, State is greater than zero. If so,control continues with 980; otherwise, control transfers to 984. At 980,control sets Test State to the next state stored in Training_Session.Control then returns to 960.

At 984, control determines whether there are any additional tests storedin Completed_Tests—that is, Test is greater than zero. If so, controlcontinues with 988; otherwise, control transfers to 992. At 988, controldecrements Test by one and sets Training_Test to the next completedtest—for example, the next test stored in Completed_Tests. Control thenreturns to 952.

At 992, control determines the overall performance of the newly trainedneural network. For example, control may use equation 5 below tocalculate the overall performance (Performance).Performance=Change_Success+Change_Failure  (5)

Control then continues with 996, where control determines if the overallperformance of the trained neutral network represents an improvement ofthe neural networks. For example, control determines if Performance isgreater than or equal to zero. If so, the newly trained neural networksare an improvement over the prior networks and control ends. IfPerformance is less than zero, the newly trained neural networks are notan improvement and control transfers to 998. At 998, control restoresthe old neural networks—in other words, control restores the neuralnetworks to the state that they were in just prior to the most recenttraining. At 998, control also sets the training status of the neuralnetworks (Networks_Trained) to the status associated with the old neuralnetworks (prior_status). Control then ends.

FIGS. 10A-10C are a flowchart depicting an example method of reweightingactions in a completed shortest path test. Although the example methodis described below with respect to the UI testing and enhancement device110, the method may be implemented in other systems. In variousimplementations, control may be performed by the shortest path module120 and/or the reweighting module 136.

Control begins at 1003 of FIG. 10A, where control determines the numberof states to be reweighted—for example, control sets State_Count to thenumber of steps stored in Test_Data. Control progresses with 1006, wherecontrol sets the state to be reweighted (Reweight_State) to the firststate of the last completed test—the first step stored in Test_Data.Control then continues with 1009.

At 1009, control determines whether the state to be reweighted includeslinks. If so, control progresses to 1012; otherwise control transfers to1015 of FIG. 10B. At 1012, control determines whether the neuralnetworks in the neural network module 144 are trained. If so, controlprogresses to 1018; otherwise control transfers to 1021. At 1018, foreach link in the state to be reweighted, control uses the neuralnetworks in the neural network module 144 to estimate the probabilitythat the link will result in the goal text and then stores the estimatedprobabilities—for example, in Reweight_State. Control then progresses to1015 of FIG. 10B.

At 1021, for each link in the state to be reweighted, control assigns arandom probability that the link will result in the goal and then storesthe random probabilities—for example, in Reweight_State. In someimplementations, control may use the Monte Carlo module 142 to generatethe random probabilities. Control then progresses to 1015 of FIG. 10B.

At 1015 of FIG. 10B, control determines whether the state to bereweighted (Reweight_State) includes a data entry field. If so, controlcontinues with 1024; otherwise control transfers to 1027 of FIG. 10C. At1024, control determines whether the state to be reweighted(Reweight_State) includes data. If so, control continues with 1030;otherwise, control transfers to 1027 of FIG. 10C.

At 1030, control determines if the neural networks are trained. If so,control continues with 1033; otherwise, control transfers to 1036. At1033, for each possible data entry field and data combination in thestate to be reweighted, control uses the neural networks in the neuralnetwork module 144—for example, the data match neural network 152—toestimate the probability that the data entry field matches the data(P_match). Control continues with 1039 where, for each data entry fieldin the state to be reweighted, control uses neural networks in theneural network module 144—for example, the data choice neural network154—to estimate the probability that entering data into the entry fieldwill result in the goal text (P_choice). Control progresses to 1042.

At 1036, for each entry field and each piece of data in the state to bereweighted, control assigns a random probability that the entry fieldmatches the data (P_match). Control progresses to 1045 where, for eachentry in the state to be reweighted, control assigns a randomprobability that entering data into the data entry field will result inthe goal (P_choice). In some implementations, control may use the MonteCarlo module 142 to generate random probabilities for P_match andP_choice. Control then progresses to 1042.

At 1042, control calculates a combined probability for each data entryfield and data combination. For example, control may average P_match andP_choice for each data entry field and data combination in the state tobe reweighted. Control stores the combined probabilities—for example, inReweight_State. Control then continues with 1027 of FIG. 10C.

At 1027 of FIG. 10C, control determines whether the state to bereweighted includes submits. If so, control progresses to 1048;otherwise, control transfers to 1051. At 1048, control determineswhether the neural networks are trained. If so, control progresses to1054; otherwise, control transfers to 1057. At 1054, for each submit inthe state to be reweighted, control uses neural networks in the neuralnetwork module 144—for example, the submit neural network 150—toestimate the probability that executing the submit will result in thegoal text and then stores the estimated probabilities—for example, inReweight_State. Control then progresses to 1051.

At 1057, for each submit in the state to be reweighted, control assignsa random probability that the submit will result in the goal text andthen stores the random probabilities—for example, in Reweight_State. Insome implementations, control may use the Monte Carlo module 142 togenerate the random probabilities. Control then progresses to 1051.

At 1051, control determines whether the neural networks in the neuralnetwork module 144 are trained. If so, control continues with 1060;otherwise, control transfers to 1063. At 1060, control uses the actionweight neural network 156 to determine a probability weighting for eachaction type—for example, links, data entry fields, and submits. Controlcontinues with 1066. At 1063, control assigns a random probabilityweighting to each action type—for example, links, entry fields, andsubmits. In some implementations, control may use the Monte Carlo module142 to determine the random probability weightings. Control continueswith 1066.

At 1066, control determines and stores an updated weighted probabilityfor each link, data entry field and data combination, and submit in thestate to be reweighted—in other words, for each action inReweight_State. Control may use equations 3 and 4, as previouslydescribed, to calculate each weighted probability. Control thenprogresses to 1069.

At 1069, control stores the reweighted state—for example, control mayset the state in Test_Data that corresponds to the reweighted state toReweight_State. At 1069, control also decrements State_Count by one.Control continues with 1072 where control determines whether there areany additional states to be reweighted—that is, whether State_Count isgreater than zero. If so, control continues with 1075 where control setsthe state to be reweighted to the next state stored in the lastcompleted test—for example, control sets Reweight_State to the nextstate stored in Test_Data. Control then returns to 1009 of FIG. 10A. Ifcontrol determines that there are no additional states to be reweighted,control ends.

FIG. 11 is a flowchart depicting an example method of building a new UIsession based on a selected action. Although the example method isdescribed below with respect to the UI testing and enhancement device110, the method may be implemented in other systems. In variousimplementations, control may be performed by the shortest path module120 and/or the session creation module 138.

Control begins at 1104 upon selection of the highest probability actionin a completed shortest path test. At 1104, control sets a current statevariable (Current State) to the step in the completed test that isassociated with the selected action. At 1104, control also initializes astack to temporarily hold a state sequence (Temp Sequence) to Null.Control then progresses to 1108, where control pushes the current state(Current State) onto the temporary stack (Temp Sequence). Controlcontinues with 1112, where control determines whether the current stateis the initial state of the completed test. If so, control progresses to1116; otherwise, control transfers to 1120.

At 1120, control sets the current state to the step prior to the currentstate—for example, the step prior to the current state in Test_Data.Control then pushes the current state (Current_State) and the actionassociated with the current state onto the temporary stack(Temp_Sequence). Control then returns to 1112.

At 1116, control initializes the state sequence for the new session—forexample, control sets Test_Data to Null, Test_Distance to zero, Steps tozero, and, for each variable in the configuration data, Used to False.Control progresses to 1124, where control loads the initial URL and setsthe current position to the center of the UI. Control continues with1128, where control pops the top entry from the temporary stack(Temp_Sequence) and adds it to the test data (Test_Data). At 1128,control also sets the action associated with the entry to the currentaction (Current Action). Control continues with 1132.

At 1132, control determines the type of the current action. If the typeis a submit, control progresses to 1136; if the type of the currentaction is a link, control transfers to 1152; if the type of the currentaction is a data entry field, control transfers to 1156. At 1136,control determines the nearest point of the UI element associated withthe submit. Control then calculates the action distance(Action_Distance) as the Euclidean distance between Current_Position andthe nearest point of the UI element associated with the submit. Controlthen continues with 1144, where control executes the submit—for example,control adds clicks on the UI element associated with the submit—andsets the current position (Current_Position) to the nearest point of theUI element associated with the submit. Control then progresses to 1148.

At 1152, control determines the nearest point of the UI elementassociated with the link. Control then calculates the action distance(Action_Distance) as the Euclidean distance between Current_Position andthe nearest point of the UI element associated with the link. Controlthen continues with 1160, where control executes the link—for example,control triggers a clicker tap on the UI element associated with thelink—and sets the current position (Current_Position) to the nearestpoint of the UI element associated with the link. Control thenprogresses to 1148.

At 1156, control determines the nearest point of the data entry field.Control then calculates the action distance (Action_Distance) as theEuclidean distance between Current_Position and the nearest point ofdata entry field. Control then continues with 1164, where control entersthe variable associated with the action into the data entry field; marksthe variable as used—sets Used to True—and sets Current_Position to thenearest point of the data entry field. Control then progresses to 1148.

At 1148, control updates the test distance and the number of steps. Forexample, control adds Action_Distance to Test_Distance and incrementsStep by one. Control then continues with 1168. At 1168, controldetermines whether the temporary stack (Temp_Sequence) is empty—forexample, the size of Temp_Sequence is zero. If so, control continueswith 1172; otherwise, control returns to 1128. At 1172, control adds thetest distance to the stored state sequence—for example, control addsTest_Distance to Test_Data. Control then ends.

FIGS. 12A-12H are graphical representations of a process of determininga path to a goal in an example UI. Specifically, FIGS. 12A-12H depict anexample process of determining a path to complete a login process in theexample UI. FIG. 12A is a rendering of an initial state 1202 of theexample UI. The process begins by setting the current position to thecenter of the screen 1204. Next, the initial state 1202 of the UI isanalyzed.

FIG. 12B is a graphical representation of the results of the analysis ofthe initial state 1202 of the UI: for example, an identification of allof the possible actions that may be taken in the initial state 1202 ofthe UI. Specifically, the analysis identified a first link 1206, asecond link 1207, a third link 1208, and a fourth link 1209. Theanalysis also identified a first data entry field 1212, a second dataentry field 1213, and a submit button 1215 labeled “Login.” The arrowsemanating from the center of the screen 1204 represent the distance fromthe current position to each identified action in the initial state1202.

In FIG. 12C, the action selected as the first step in the shortest pathis entering the text “testuser” into the first data entry field 1212.This selection may be made, as described above, based on weightedprobabilities from trained neural networks. The distance between thecurrent position—the center of the screen—and the location of the firstdata entry field is a first step distance 1218 that is stored as thedistance of the shortest path. Performing the selected action results inthe UI transitioning from the initial state 1202 to a second state 1219and changing the current position to the location of the first dataentry field 1212. Next, the second state 1219 of the UI is analyzed.

FIG. 12D is a graphical representation of the results of an analysis ofpossible actions that may be taken in the second state 1219 of the UI.Specifically, the analysis identified the first link 1206, the secondlink 1207, the third link 1208, the fourth link 1209, the second dataentry field 1213, and the submit button 1215 as possible actions for thesecond step. The arrows emanating from the first data entry field 1212represent the distances from the current position to each identifiedaction in the second state 1219 of the UI.

In FIG. 12E, the action selected as the second step is entering textinto the second data entry field 1213—represented as “******”. Thedistance between the current position—the location of the first dataentry field 1212—and the location of the second data entry field 1213 isa second step distance 1220 that is added to the stored current pathdistance. Performing the selected action in the second state 1219results in the UI transitioning to a third state 1222 and changing thecurrent position to the location of the second data entry field 1213.Next, the third state 1222 of the UI is analyzed.

FIG. 12F is a graphical representation of the results of an analysis ofthe third state 1222 of the UI—in other words, all of the possibleactions that may be taken in the third state 1222 of the UI.Specifically, the analysis identified the first link 1206, the secondlink 1207, the third link 1208, the fourth link 1209, and the submitbutton 1215 as possible actions for the third step of the shortest path.The arrows emanating from the second data entry field 1213 represent thedistance from the current position to each identified action in thethird state 1222.

In FIG. 12G, the action selected as the third step is clicking thesubmit button 1215 labeled “Login.” The distance between the currentposition—the location of the second data entry field 1213—and thelocation of the submit button 1215 labeled “Login.” is a third stepdistance 1224 that is added to the stored current path distance.Performing the selected action in the third state 1222 results in the UItransitioning to a fourth state 1226 shown in FIG. 12H and changing thecurrent position to the location of the submit button 1215 labeled“Login.”

As shown in FIG. 12H, an analysis of the fourth state 1226 of the UIresults in the identification of text 1230. The text 1230 matches thegoal text (for example, “Welcome”), which indicates that the goal hasbeen reached. The distance from the current position—the former locationof the submit button 1215 labeled “Login”—to the location of the text1230 is a fourth step distance 1232 that is added to the current pathdistance to determine the total distance of the current path. This pathled to a successful outcome. Additional paths may be investigates todetermine whether a successful path with a shorter total distance can befound.

User Interface Enhancement

FIG. 13 is a flowchart depicting an example method of generating promptsand reinforcements for a UI using eye tracking experiments. Although theexample method is described below with respect to the UI testing andenhancement device 110, the method may be implemented in other devicesand/or systems. In various implementations, control may be performed bythe UI enhancement module 122 and the shortest path module 120.

Control begins at 1304, where control obtains an initial URL for a UI, agoal for the UI, and a reinforcement percentage. The reinforcementpercentage is a threshold used to determine if a step of the shortestpath to the goal requires user reinforcement. For example, thereinforcement percentage indicates how often users may deviate from astep of the shortest path before the step is determined to requirereinforcement. Control continues with 1306. At 1306, control determinesthe shortest path in the UI to the goal. In some implementations,control may perform the method disclosed in FIGS. 7A and 7B, aspreviously described. In other implementations, control may load apreviously stored shortest path associated with the goal for the UI—forexample, from the storage device 112. Control then progresses to 1308.

At 1308, control determines a reinforcement distance for each URL of theUI that is encountered along the shortest path. The reinforcementdistance is the maximum distance that a user may deviate from theapplicable shortest path step before control determines that the stepmay require reinforcement. Control may perform the method described inFIG. 14 to determine the reinforcement distance for each URL. Controlthen continues with 1312.

At 1312, control determines, based on the determined reinforcementdistance for each URL of the UI, the steps of the shortest path that mayrequire reinforcement. For example, control may perform the methoddisclosed in FIG. 21 to determine which steps may require reinforcement.Control continues with 1316.

At 1316, control loads the first step of the shortest path. Control alsoinitiates UI_Enhancement to Null. UI_Enhancement is used to storeprompts and reinforcements related to the UI that may be used to aid auser in reaching the goal. Control then progresses to 1320.

At 1320, control determines whether the step includes a link. If so,control continues with 1324; otherwise, control transfers to 1328. At1324, control obtains the DOM element that corresponds to the includedlink. Control continues with 1332 where control determines, based on theobtained DOM element, whether the link is represented by a button. Ifso, control continues with 1336; otherwise, control transfers to 1340.

At 1336, control generates a prompt that instructs a user to click onthe button associated with the link. For example, control may set theprompt to “Click on <<button>>.” In the generated prompt, <<button>> istext that describes the button—for example, the name of the button ortext associated with the button. Control may obtain the name of, or thetext associated with, the button from the obtained DOM element. Controlthen progresses to 1344, as described below.

At 1340, control generates a prompt that instructs a user to select anoption from a menu. For example, control may set the prompt to “Select<<option>> from <<menu>>.” In the generated prompt, <<option>> and<<menu>> are texts that describe the option and the menu, respectively.Control may obtain the text from the obtained DOM element. Controlcontinues with 1344, as described below.

Returning to 1328, control determines whether the step includes asubmit. If so, control continues with 1348; otherwise, controldetermines that the step includes a data entry field and controltransfers to 1352. At 1348, control obtains the DOM element thatcorresponds to the submit included in the step. Control continues with1356, where control generates a prompt that instructs a user to click onthe submit. For example, control may set the prompt to “Click on<<submit>>.” In the generated prompt, <<submit>> is text that describesthe submit—for example, the name of the submit or text associated withthe submit. Control may obtain the name of, or the text associated with,the submit from the obtained DOM element. Control then progresses to1344, as described below.

Returning to 1352, control obtains the DOM element that corresponds tothe data entry field. Control continues with 1360 where controldetermines, based on the obtained DOM element, whether the data entryfield is a text entry field. If so, control continues with 1364;otherwise, control may infer that the entry field is a list of one ormore options and transfer to 1368. At 1364, control generates a promptthat instructs a user to enter text into the entry field. For example,control may set the prompt to “Enter your data in <<field>>.” In thegenerated prompt, <<field>> is text that describes the entry field—forexample, the name of the entry field or text associated with the entryfield. Control may obtain the name of, or the text associated with, theentry field from the obtained DOM element. Control then progresses to1344.

Returning to 1368, control generates a prompt that instructs a user toselect an option from the list of options. For example, control may setthe prompt to “Select <<option>> from the options in <<field>>.” In thegenerated prompt, <<field>> is text that describes the entry field—forexample, the name of the entry field or text associated with the entryfield. Control may obtain the name of, or the text associated with, theentry field from the obtained DOM element. Control continues with 1344.

Returning to 1344, control determines whether the step requires userreinforcement based on the received reinforcement percentage. Forexample, control may use inequality 6 below to determine whether thestep requires reinforcement.

$\begin{matrix}{{\frac{{Count}_{reinforce}}{{Count}_{total}}*100} > {{Reinforcement}\;\%}} & (6)\end{matrix}$

Count_(reinforce) represents the number of times the step is referencedas potentially requiring reinforcement—each occurrence of an eyetracking data sample deviating from the step by more than thecorresponding reinforcement distance. Count_(total) is the total numberof times that an eye tracking sample was determined to correspond to thestep. If inequality 6 is true, control determines that the step requiresreinforcement and control continues with 1372; otherwise, controltransfers to 1376.

At 1372, control marks the component of the UI associated with theobtained DOM element as requiring additional reinforcement. In someimplementations, the UI component may be highlighted while the prompt ispresented to a user. In other implementations, the location of the UIcomponent may be provided to a user. Control then progresses to 1376.

At 1376, control stores the shortest path step, the DOM element, thegenerated prompt, and any additional reinforcement in UI_Enhancement.Control continues with 1380.

At 1380, control determines whether the shortest path includes a nextstep. If so, control continues with 1384; otherwise, control transfersto 1388. At 1384, control loads the next step in the shortest path andthen returns to 1320.

At 1388, control outputs UI_Enhancement. Control then ends. The UIenhancements generated may be used to generate a new UI that guides auser to the goal. For example, the generated prompts may be incorporatedinto the UI. The data contained in UI_Enhancement may also be used by anautomated system to provide a user with prompts that guide the user to arequested or suggested goal.

In some implementations, control may only generate a prompt in responseto determining that a step of the shortest path requires reinforcement.In other words, control may only implement elements 1320, 1324, 1328,1332, 1336, 1340, 1348, 1352, 1356, 1360, 1364, and 1368 in response tocontrol determining (as at 1344) that the step requires userreinforcement based on the received reinforcement percentage. Forexample, control may use inequality 6 above to determine whether thestep requires reinforcement.

FIG. 14 is a flowchart depicting an example method for determining areinforcement distance for each URL of a UI. Although the example methodis described below with respect to the UI testing and enhancement device110, the method may be implemented in other devices and/or systems. Invarious implementations, control may be performed by the reinforcementmodule 160 and/or the distance module 164.

Control begins at 1404 where control obtains a shortest path for a goal.In some implementations, control may load a shortest path that has beenpreviously determined and stored—for example in the storage device 112.In other implements, control may perform the method disclosed in FIGS.7A and 7B. Control continues with 1408 where control obtains testingparameters—for example, a maximum number of evolutions that the systemis permitted to take (Max_Evolutions) and a set of fitness weightingvalues. Control then progresses to 1412.

At 1412, control loads eye tracking experiments that correspond to theUI and the goal. The eye tracking experiments include tracking data of auser navigating the UI. The tracking data is captured at a fixedsampling frequency—for example, every two seconds. Each eye trackingexperiment groups the tracking data into pages based on the page of theUI being navigated by the user, in the order that they were viewed bythe user. In other words, each page of an eye tracking experimentrepresents the movements of a user during a single state of the UI. Eachdata point in the tracking data includes the page of the UI (URL) at thetime of capture and a set of coordinates that indicate a location on thescreen of the UI. Control continues with 1416.

At 1416, control calculates the differences between the eye trackingexperiments and the shortest path. For example, control determines thedistances between the tracking data and points along the shortest path.Control may perform the method disclosed in FIG. 15 to calculate thedistances. Control then progresses to 1420.

At 1420, control calculates the median successful distance for each URLincluded in the eye tracking experiments marked as successes—experimentsthat resulted in the user reaching the goal. Control may perform themethod disclosed in FIG. 17 to determine the median successfuldistances. Control then progresses to 1424.

At 1424, control calculates the median failure distance for each URLincluded in eye tracking experiments marked as failures. An experimentmay be marked as a failure if the user did not reach the goal or if thetracking data deviated from the shortest path. Control may perform themethod disclosed in FIG. 18 to determine the median failure distances.Control continues with 1428.

At 1428, control sets boundary values for each URL based on thecorresponding median success and failure distances. For example, controlsets an upper bound for the URL to the median success distance and setsa lower bound for the URL to the median failure distance. The upper andlower bounds are used to generate guesses of distance of deviations fromthe shortest path that require reinforcement. Control also initiates thenumber evolutions performed (Evolutions) to zero. Control thenprogresses to 1432.

At 1432, for each URL, control generates a random distance based on theboundary values for the URL. For example, control may use the MonteCarlo module 142 to generate a random value between the upper bound forthe URL and the lower bound for the URL. Control continues with 1436,where control tests the generated values for the URL. For example,control may perform the method disclosed in FIG. 19 to test thegenerated distances. Control then progresses to 1440.

At 1440, control determines whether the number of evolutions performedis less than the maximum number of allowed evolutions. If so, controlcontinues with 1444; otherwise, control transfers to 1448. At 1444,control increments Evolutions and returns to 1432.

At 1448, control calculates a fitness value for each randomly generatedreinforcement distance. Control may use equation 7 below to calculateeach fitness value.

$\begin{matrix}{{FitnessValue} = \frac{\begin{pmatrix}{{W_{1}*{SuccessCount}} -} \\\begin{matrix}{{W_{2}*{FalsePositiveCount}} -} \\{W_{3}*{FalseNegativeCount}}\end{matrix}\end{pmatrix}}{TotalCount}} & (7)\end{matrix}$

W₁, W₂, and W₃ are the fitness weighting values obtained in 1408.SuccessCount represents the number of times that a prediction, based onthe randomly generated reinforcement distance for the URL, correctlypredicted the outcome of an eye tracking experiment. FalsePositiveCountrepresents the number of times that a prediction, based on the randomlygenerated reinforcement distance for the URL, incorrectly predicted thata successful eye tracking experiment was a failure. FalseNegativerepresents the number of times that a prediction, based on the randomlygenerated reinforcement distance for the URL, incorrectly predicted thata failed eye tracking experiment was a success. Control continues with1452.

At 1452, control performs cluster analysis on a plot of the determinedfitness values versus the associated randomly generated distances. Insome implementations, control uses the density-based spatial clusteringof applications with noise (DBSCAN) algorithm to perform the clusteranalysis. In other implementations, control uses the mean shifttechnique to perform the cluster analysis. In yet other implementations,control may use any suitable cluster analysis technique or algorithm toanalyze the plot of fitness values versus randomly generated distances.Control continues with 1456.

At 1456, control determines whether more than one cluster was identifiedfor a single URL during the cluster analysis. If so, control continueswith 1460; otherwise, control transfers to 1464. At 1460, controlupdates the boundary value for each URL based on the most successfulcluster of the URL. For example, for each URL, control sets the upperbound for the URL to the largest distance in the most successful clusterand sets the lower bound for the URL to the smallest distance in themost successful cluster. Control also resets the number of performedevolutions to zero. Control then returns to 1432.

Returning to 1464, control outputs the reinforcement distances. Forexample, for each URL, control outputs the reinforcement distanceassociated with the URL with the highest fitness value. Control thenends.

FIG. 15 is a flowchart depicting an example method for determining thedistances between eye tracking experiments and points along a loadedshortest path. Although the example method is described below withrespect to the UI testing and enhancement device 110, the method may beimplemented in other devices and/or systems. In various implementations,control may be performed by the reinforcement module 160 and/or thedistance module 164.

Control begins with 1502. At 1502, control obtains eye trackingexperiments and a shortest path. In some implementations, control mayload previously stored eye tracking experiments and/or a shortestpath—for example, from the storage device 112. In other implementations,control may use eye tracking experiments and/or a shortest path that arealready loaded. Control continues with 1504, where control sets variableCurrent_Experiment to the first loaded eye tracking experiment. Controlcontinues with 1508 where control sets Current_Page to the first pagestored in Current_Experiment. At 1508, control also loads the steps ofthe shortest path into a temporary queue (Temp_Path). Control thenprogresses to 1512.

At 1512, control determines the distances for the current page using thetemporary queue. For example, control may perform the method disclosedin FIG. 16 to load Current_Page and calculate the distances based on theshortest path steps in Temp_Path. Control then progresses to 1516. At1516, control determines whether the tracking data associated with thecurrent page deviated from the shortest path—for example, whether Resultequals deviation. If so, control continues with 1520; otherwise controltransfers to 1524. At 1520, control logs the path deviation for thecurrent page. For example, control stores the deviation in Current_Page.Control continues with 1524.

At 1524, control logs the determined distances and associated data forthe current page—for example, control may add Result to Current_Page.Control continues with 1526, where control updates Current_Experimentbased on the data associated with data stored in Current_Page. Controlprogresses to 1528, where control determines whether Current_Experimentincludes additional pages. If so, control continues with 1532;otherwise, control transfers to 1536. At 1532, control sets Current_Pageto the next page stored in Current_Experiment. Control then returns to1512.

At 1536, control determines whether any of the pages inCurrent_Experiment include a deviation. If so, control continues with1540; otherwise, control transfers to 1544. At 1540, control sets thestatus associated with the distances (Distance_Status) stored in thecurrent experiment to failure. Control then continues with 1548. At1544, control sets Distance_Status stored in the current experiment tothe status of the current experiment (Current_Experiment)—for example,either success or failure. Control continues with 1548.

At 1548, control logs the status associated with the distances stored inthe current experiment. For example, control may add Distance_Status toCurrent_Experiment. Control progresses to 1550, where control updatesthe loaded eye tracking experiments based on the data associated withthe current experiment—in other words, the data stored inCurrent_Experiment. Control then continues with 1552.

At 1552, control determines whether the loaded eye tracking experimentsinclude another experiment. If so, control continues with 1556;otherwise, control transfers to 1560 where control saves the loadedexperiments and control ends. At 1556, control sets Current_Experimentto the next loaded experiment. Control then returns to 1508.

FIG. 16 is a flowchart depicting an example method for determining thedistances between eye tracking data and points along a loaded shortestpath for a page of the UI. Although the example method is describedbelow with respect to the UI testing and enhancement device 110, themethod may be implemented in other devices and/or systems. In variousimplementations, control may be performed by the reinforcement module160 and/or the distance module 164.

Control begins with 1604 upon receipt of eye tracking data associatedwith the page of the UI and a queue that contains steps of the shortestpath—for example, Current_Page and Temp_Path from 1512 of FIG. 15. At1604, control determines the URL associated with the eye tracking dataand sets Result to Null. Control continues with 1608, where controlpulls eye tracking samples in coordinate form from the eye trackingdata—for example, Current_Page. Control continues with 1612, wherecontrol plots the eye tracking samples on a grid and generates a vectorthat represents the samples. Control progresses to 1616.

At 1616, control loads a shortest path step—for example, control pops astep from Temp_Path. Control then compares the URL associated with theeye tracking data and the URL associated with the shortest path step.Control progresses to 1620. If the URLs match, control continues with1624; otherwise, control determines that the eye tracking data deviatesfrom the shortest path and control transfers to 1628. At 1628, controlsets the result for the overall determination (Result) to deviation andcontrol continues with 1632. At 1632, control outputs Result and ends.

At 1624, control plots the start and end coordinates of the shortestpath step on the grid. Control then generates a shortest path vectorbetween the start and end points of the step. Control progresses to1636, where control determines whether there is another step in theshortest path—for example, whether the size of Temp_Path is greater thanzero. If so, control progresses to 1640; otherwise, control transfers to1644.

At 1640, control compares the URL associated with the next step in theshortest path with the URL associated with the eye tracking data—forexample, control peeks at the next step in Temp_Path without removingthe step from the queue. Control then progresses to 1648. If the URLsmatch, control continues with 1652; otherwise, control transfers to1644. At 1652, control loads the next shortest path step—for example,control pops a step off of Temp_Path. Control then updates the grid andthe shortest path vector based on the next shortest path step. Controlthen returns to 1636.

At 1644, control plots points that correspond to the tracking samplesalong the shortest path vector. For example, control plots one point foreach plotted eye tracking sample equidistantly along the shortest pathvector, beginning at the start of the shortest path vector. Controlcontinues with 1656, where control sets the current point to the pointat the start of the shortest path vector. Control then progresses to1660.

At 1660, control calculates the Euclidean distance between the currentpoint and the eye tracking sample that corresponds to the current point.At 1664, control logs the coordinates of both the current point and thecorresponding eye tracking sample, as well as the distance between thetwo. For example, control may add these values to Result. Control thenprogresses to 1668 where control determines whether the current point isat the end of shortest path vector. If so, control continues with 1632,where control outputs Results and ends; otherwise, control transfers to1672. At 1672, control sets the current point to the next point on theshortest path vector and returns to 1660.

FIG. 17 is a flowchart depicting an example method of determining themedian successful distance for each URL in a set of eye trackingexperiments based on data stored in the eye tracking experiments.Although the example method is described below with respect to the UItesting and enhancement device 110, the method may be implemented inother devices and/or systems. In various implementations, control may beperformed by the reinforcement module 160 and/or the distance module164.

Control begins with 1702. At 1702, control obtains eye trackingexperiments. In some implementations, control may load previously storedeye tracking experiments—for example, from the storage device 112. Inother implements, control may use eye tracking experiments that arealready loaded. Control continues with 1704, where control sets thecurrent experiment to the first experiment of the eye trackingexperiments and Successful_Temp to Null. Control uses Successful_Temp tostore each successful distance and associated URL. Control thenprogresses to 1708.

At 1708, control loads the data and status of the current experiment.Control continues with 1712, where control determines whether thecurrent experiment is marked as a success—in other words, the test userreached the goal. If so, control continues with 1716; otherwise, controltransfers to 1720. At 1716, control loads all of the distancesassociated with the first URL of the current experiment. Control thenprogresses to 1724, where control adds each distance associated with theURL along with the URL to Successful_Temp. In other words, each distancestored in Successful_Temp is linked to a URL. Control continues with1728, where control determines whether there is another URL in thecurrent experiment. If so, control continues with 1732, where controlloads all of the distances associated with the next URL in the currentexperiment and then control returns to 1724; otherwise, controltransfers to 1720.

At 1720, control determines whether there is another eye trackingexperiment. If so, control continues with 1736; otherwise, controltransfers to 1740. At 1736, control loads the data and status of thenext experiment and then returns to 1708. At 1740, control determinesthe median distance stored in Successful_Temp for each URL. Controlcontinues with 1744, where control outputs the median successfuldistance for each URL in the eye tracking experiments. Control thenends.

FIG. 18 is a flowchart depicting an example method of determining themedian failure distance for each URL in a set of eye trackingexperiments based on data stored in the eye tracking experiments.Although the example method is described below with respect to the UItesting and enhancement device 110, the method may be implemented inother devices and/or systems. In various implementations, control may beperformed by the reinforcement module 160 and/or the distance module164.

Control begins with 1802, where control obtains the eye trackingexperiments. In some implementations, control may load previously storedeye tracking experiments—for example, from the storage device 112. Inother implements, control may use eye tracking experiments that arealready loaded. Control continues with 1804, where control sets thecurrent experiment to the first experiment of the eye trackingexperiments and Failure_Temp to Null. Control uses Failure_Temp to storeeach failure distance and associated URL. Control then progresses to1808.

At 1808, control loads the data and status of the current experiment.Control continues with 1812 where control determines whether the currentexperiment is marked as a failure. An experiment may be marked as afailure in response to either the test user failing to reach the goal orthe tracking data deviating from the shortest path. If the experiment ismarked as a Failure, control continues with 1816; otherwise, controltransfers to 1820. At 1816, control loads all of the distancesassociated with the first URL of the current experiment. Control thenprogresses to 1824, where control adds each distance associated with theURL along with the URL to Failure_Temp. In other words, each distancestored in Failure_Temp is linked to a URL. Control continues with 1828,where control determines whether there is another URL in the currentexperiment. If so, control continues with 1832, where control loads allof the distances associated with the next URL in the current experimentand then control returns to 1824; otherwise, control transfers to 1820.

At 1820, control determines whether there is another eye trackingexperiment. If so, control continues with 1836; otherwise, controltransfers to 1840. At 1836, control loads the data and status of thenext experiment and then returns to 1808. At 1840, control determinesthe median distance stored in Failure_Temp for each URL. Controlcontinues with 1844, where control outputs the median failure distancefor each URL in the eye tracking experiments. Control then ends.

FIG. 19 is a flowchart depicting an example method for comparingpredictions of the outcomes of eye tracking experiments with previouslystored results. The predictions are based on a shortest path to a goaland a set of test distances that represent an acceptable deviation fromthe shortest path. Although the example method is described below withrespect to the UI testing and enhancement device 110, the method may beimplemented in other devices and/or systems. In various implementations,control may be performed by the reinforcement module 160, the distancemodule 164, and/or the prediction module 168.

Control begins with 1904, where control obtains the eye trackingexperiments, shortest path, and test distances. In some implementations,control may load previously stored eye tracking experiments, a shortestpath, and/or test distances—for example, from the storage device 112. Inother implements, control may use eye tracking experiments, a shortestpath, and/or test distances that are already loaded. Control continueswith 1908, where control sets the current experiment to the first loadedeye tracking experiment and sets Prediction to Null. Control usesPrediction to store the generated predictions for each URL. Controlcontinues with 1912, where control sets the current page to the firstpage stored in the current experiment. At 1912, control also loads theshortest path into a temporary queue (Temp_Path). Control thenprogresses to 1916.

At 1916, control generates predictions for the current page using thetest distances and the shortest path steps stored in the temporary queue(Temp_Path). For example, control may perform the method disclosed inFIG. 20 to generate the predictions. Control then progresses to 1920,where control logs the generated prediction—for example, control addsthe prediction to Prediction. Control then progresses to 1924.

At 1924, control determines whether there is another page in the currentexperiment. If so, control progresses to 1928; otherwise, controltransfers to 1932. At 1928, control sets the current page to the nextpage in the current experiment. Control then returns to 1916.

At 1932, control determines whether there is another experiment. If so,control continues with 1936; otherwise, control transfers to 1940. At1940, control outputs the generated predictions for each URL(Prediction). Control then ends.

FIG. 20 is a flowchart depicting an example method for comparing aprediction of an outcome of an eye tracking experiment, based on ananalysis of a single page of the eye tracking experiment, withpreviously stored results. Although the example method is describedbelow with respect to the UI testing and enhancement device 110, themethod may be implemented in other devices and/or systems. In variousimplementations, control may be performed by the reinforcement module160, the distance module 164, and/or the prediction module 168.

Control begins with 2004, upon receipt of a page of the eye trackingexperiment, test distances, and a queue that contains steps of theshortest path—for example, Current_Page, Test Distances, and Temp_Pathfrom 1916 of FIG. 19. At 2004, control determines and stores the statusof the experiment—such as success or failure—associated with the page ofeye tracking data. For example, control sets Status to the storedoutcome of the experiment associated with the received page. Controlalso initiates Result to Null. Control uses Result to store the resultsof the comparisons for the page of eye tracking data. Control thencontinues with 2008.

At 2008, control determines the URL associated with the page of eyetracking data. Then control determines and stores the test distanceassociated with the determined URL (Test_Dist). Control continues with2012, where control pulls eye tracking samples in coordinate form fromthe eye tracking data—for example, Current_Page. Control then progressesto 2016, where control plots the eye tracking samples on a grid andgenerates a vector that represents the samples. Control continues with2020.

At 2020, control loads a shortest path step—for example, control pops astep from Temp_Path. Control then compares the URL associated with theeye tracking data and the URL associated with the shortest path step.Control progresses to 2024. If the URLs match, control continues with2028; otherwise, control determines that the eye tracking data and theshortest path step are not associated with the same screen of the UI andcontrol ends.

At 2028, control plots the start and end coordinates of the shortestpath step on the grid. Control then generates a shortest path vectorbetween the start and end points of the step. Control progresses to2032, where control determines whether there is another step in theshortest path—for example, the size of Temp_Path is greater than zero.If so, control progresses to 2032; otherwise, control transfers to 2040.

At 2032, control compares the URL associated with the next step in theshortest path with the URL associated with the eye tracking data—forexample, control peeks at the next step in Temp_Path without removingthe step from the queue. Control then progresses to 2044. If the URLsmatch, control continues with 2048; otherwise, control transfers to2040. At 2048, control loads the next shortest path step—for example,control pops a step off of Temp_Path. Control then updates the grid andthe shortest path vector based on the step next. Control then returns to2032.

At 2040, control plots points that correspond to the plotted trackingsamples along the shortest vector. For example, control plots one pointfor each plotted eye tracking sample equidistantly along the shortestpath vector, beginning at the start of the vector. Control continueswith 2052, where control sets the current point (Current_Point) to thepoint at the start of the shortest path vector. Control then progressesto 2056.

At 2056, control calculates the Euclidean distance between Current_Pointand the eye tracking sample that corresponds to CurrentPoint(Calculated_Dist). Control progresses to 2060, where control calculatesa prediction of the outcome of the experiment based on the distancebetween tCurrent_Point and the associated tracking sample and the testdistance for the URL associated with Current_Point and tracking sample.For example, control determines if Calculated_Dist is greater thanTest_Dist. If so, control predicts that the experiment is a failure;otherwise, control predicts that the experiment is a success. Controlthen progresses to 2064.

At 2064, control compares the calculated prediction with the previouslystored status of the experiment (Status) and classifies the prediction.For example, if the prediction and Status match, control determines thatthe prediction is a success. If control predicted a failed experimentand Status indicates a success, control determines that the predictionis a false positive. Conversely, if control predicted a successfulexperiment and Status indicates a failure, control determines that theprediction is a false negative. Control then stores the determinedprediction status. For example, control adds the determined status alongwith the associated URL and test distance (Test_Dist) to Result. Controlthen progresses to 2068.

At 2068, control determines whether Current_Point is at the end of theshortest path vector. If so, control continues with 2072, where controloutputs Result and ends; otherwise, control transfers to 2076. At 2076,control sets the current point to the next point on the shortest pathvector. Control then returns to 2056.

FIG. 21 is a flowchart depicting an example method for determining thesteps of a shortest path to a goal in a UI that may requirereinforcement. The determination is based on eye tracking experimentsand predetermined reinforcement distances for each URL in the UI.Although the example method is described below with respect to the UItesting and enhancement device 110, the method may be implemented inother devices and/or systems. In various implementations, control may beperformed by the reinforcement module 160.

Control begins with 2104, where control obtains eye trackingexperiments, the shortest path, and reinforcement distances. In someimplementations, control may load previously stored eye trackingexperiments, a shortest path, and/or reinforcement distances—forexample, from the storage device 112. Control continues with 2108, wherecontrol sets the current experiment to the first loaded eye trackingexperiment and Reinforcement to Null. Control uses Reinforcement tostore information regarding which steps require reinforcement. Controlcontinues with 2112, where control sets the current page to the firstpage stored in the current experiment. At 2112, control also loads theshortest path into a temporary queue (Temp_Path). Control thenprogresses to 2116.

At 2116, control generates reinforcement data that identifies the stepsassociated with the current page that may require reinforcement based onthe eye tracking data of the current page and the reinforcementdistance. For example, control may perform the method disclosed in FIG.21 to identify the steps. Control then progresses to 2120, where controllogs the generated reinforcement data—for example, control adds the datato Reinforcement. Control then progresses to 2124.

At 2124, control determines whether there is another page in the currentexperiment. If so, control progresses to 2128; otherwise, controltransfers to 2132. At 2128, control sets the current page to the nextpage in the current experiment. Control then returns to 2116.

At 2132, control determines whether there is another experiment. If so,control continues with 2136; otherwise, control transfers to 2140. At2136, control sets the next experiment as the current experiment andthen control returns to 2112. At 2140, control outputs the reinforcementdata (Reinforcement). Control then ends.

FIG. 22 is a flowchart depicting an example method for determining stepsof a shortest path associated with a page of an eye tracking experimentthat may need reinforcement. Although the example method is describedbelow with respect to the UI testing and enhancement device 110, themethod may be implemented in other devices and/or systems. In variousimplementations, control may be performed by the reinforcement module160.

Control begins with 2204, upon receipt of a page of the eye trackingexperiment, reinforcement distances, and a queue that contains steps ofthe shortest path—for example, Current_Page, Test_Distances, andTemp_Path from 2116 of FIG. 21. At 2204, control determines the URLassociated with the page of eye tracking data. Control progresses to2208, where control determines the reinforcement distance for the URLassociated with the page of the eye tracking data and sets Reinforcementto Null. Control uses Reinforcement to store information regarding whichsteps require reinforcement. Control then continues with 2212.

At 2212, control pulls eye tracking samples in coordinate form from theeye tracking data—for example, Current_Page. Control then progresses to2216, where control plots the pulled eye tracking samples on a grid andgenerates a vector that represents the samples. Control continues with2220.

At 2220, control loads a shortest path step—for example, control pops astep from Temp_Path. Control then compares the URL associated with theeye tracking data and the URL associated with the loaded shortest pathstep. Control progresses to 2224. If the URLs match, control continueswith 2228; otherwise, control determines that the eye tracking data andthe shortest path step are not associated with the same screen of the UIand control ends.

At 2228, control plots the start and end coordinates of the loadedshortest path step on the grid. Control then generates a shortest pathvector between the start point and the end point of the step. Controlprogresses to 2232, where control determines whether there is anotherstep in the shortest path—for example, the size of Temp_Path is greaterthan zero. If so, control progresses to 2236; otherwise, controltransfers to 2240.

At 2232, control compares the URL associated with the next step in theshortest path with the URL associated with the eye tracking data—forexample, control peeks at the next step in Temp_Path without removingthe step from the queue. Control then progresses to 2244. If the URLsmatch, control continues with 2248; otherwise, control transfers to2240. At 2248, control loads the next shortest path step—for example,control pops a step from Temp_Path. Control updates the grid and theshortest path vector based on the loaded shortest path step. Controlthen returns to 2232.

At 2240, control plots points that correspond to the tracking samplesalong the shortest vector. For example, control plots one point for eachplotted eye tracking sample equidistantly along the shortest pathvector, beginning at the start of the vector. Control continues with2252, where control sets the current point to the point at the start ofthe shortest path vector. Control then progresses to 2256.

At 2256, control calculates the Euclidean distance (Calculated_Dist)between the current point and the eye tracking sample that correspondsto the current point. Control progresses to 2260, where controldetermines the shortest path step associated with the current point.Control then continues with 2264.

At 2264, control determines whether the calculated distance(Calculated_Dist) is greater than the reinforcement distance(Reinforce_Dist). If so, control determines that the step may needreinforcement and control continues with 2268; otherwise, controltransfers to 2272. At 2268, control adds a reference to the shortestpath step associated with the current point and an indication that thestep may require reinforcement to Reinforcement. Control then continueswith 2276. At 2272, control adds a reference to the shortest path stepassociated with the current point and an indication that the step doesnot require reinforcement to Reinforcement. Control then progresses to2276.

At 2276, control determines whether the current point is at the end ofthe shortest path vector. If so, control continues with 2280; otherwise,control transfers to 2284. At 2284, control sets the current point tothe next point on the shortest path vector. Control then returns to2256. At 2280, control outputs the determined reinforcement results forthe shortest path steps associated with the eye tracking data—forexample, control outputs Reinforcement. Control then ends.

FIGS. 23A-23E are visual representations of the process of generating anexample eye tracking vector and an example shortest path vector for apage of a UI. FIG. 23A shows an example page 2302 of a UI. First throughtenth eye tracking samples 2310-2319 of an eye tracking experiment thatcorrespond to the page 2302 are plotted on the UI. Each eye trackingsample represents the point of focus of a test user navigating the UI atthe time the sample was captured. As an example, the eye trackingsamples 2310-2319 may represent the location of a mouse cursor that wascaptured once every two seconds. An eye tracking vector 2320 thatincludes each of the plotted eye tracking samples 2310-2319 isgenerated. The eye tracking vector may be piecewise linear or smoothedand is representative of the path taken by the test user during the eyetracking experiment.

In FIG. 23B, a first shortest path point 2330 that corresponds to thebeginning of a first shortest path step and a second shortest path point2331 that corresponds to the end of the first shortest path step areplotted on the page 2302. In FIG. 23C, a third shortest path point 2332that corresponds to the end of a second shortest path step that began atthe second shortest path point 2331 is plotted on the page 2302. In FIG.23D, a fourth shortest path point 2333 that corresponds to the end of athird shortest path step that began at the third shortest path point2332 is plotted on the page 2302.

In FIG. 23E, a shortest path 2335 is shown on the page 2302. Theshortest path 2335 is a vector between the first shortest path point2330, the second shortest path point 2331, the third shortest path point2332, and the fourth shortest path point 2333. The shortest path 2335represents a path on the page 2302 of the UI that corresponds to theshortest path.

First through tenth shortest path points 2340-2349 are equidistantlyplotted along the shortest path 2335. Each point of the shortest pathpoints 2340-2349 corresponds to a similarly numbered and labeled sampleof the plotted eye tracking samples 2310-2319 (that is, the final digitsmatch). For example, the first shortest path point 2340 corresponds tothe first eye tracking sample 2310 and the second shortest path point2342 corresponds to the second eye tracking sample 2311.

The shortest path vector points 2340-2349 and the corresponding the eyetracking samples 2310-2319 depicted in FIG. 23E may be used to determinethe reinforcement distance for the page 2302. For example, respectivedistances between the shortest path vector points 2340-2349 and thecorresponding eye tracking samples 2310-2319 may radiate how far thetest user deviated from the shortest path. Depending on the outcome ofthe eye tracking experiment, these distances may be used to determinethe median successful distance or the median failure distance for thepage 2302. In addition, these distances may be used to test the randomlygenerated reinforcement distances for the page 2302.

Customer Migration

FIG. 24 is a block diagram of an example customer migration system 2400.The customer migration system 2400 includes an interactive voiceresponse (IVR) platform 2402. The IVR platform 2402 includes an IVRdevice 2404 and an IVR server 2408. The customer migration system 2400may also include a web application server 2412 and a cobrowse server2416.

An organization may use the IVR device 2404 to implement a customerservice number that a user can call to request information or perform atask. For example, the user may use a telephone 2420 to access the IVRdevice 2404 via a public switched telephone network (PSTN) 2422, Voiceover Internet Protocol (VoIP), or a combination of the two. The IVRdevice 2404 receives a voice request from the user and communicates withthe IVR server 2408 to determine an appropriate response to the user'srequest.

In some implementations, the IVR device 2404 may use the speechrecognition module 2424 to translate verbal inputs provided by the userover the telephone 2420 to a voiceXML browser 2426. In otherimplementations, the IVR device 2404 may outsource the translation ofthe verbal inputs to a remote service. For example, the IVR device 2404may transmit the verbal inputs received by the voiceXML browser 2426 tothe remote service and receive a translation or transcription of theverbal inputs from the remote service. The voiceXML browser 2426provides the translation to a voiceXML application 2428 of the IVRserver 2408.

The voiceXML application 2428 may determine that the user can completethe request using a web application 2430 hosted at the web applicationserver 2412. In response to determining that the user can complete therequest using the web application 2430, the voiceXML application 2428requests a cobrowse configuration from the cobrowse configuration module2432. The cobrowse configuration module 2432 provides a cobrowseconfiguration associated with the user's request to the voiceXMLapplication 2428. The cobrowse configuration is used to establish acobrowse session on the cobrowse server 2416. A cobrowse sessionprovides the voiceXML application 2428 with the current state of the webapplication 2430 accessed by the user—in other words, the current stateof a UI as viewed by the user.

Upon receiving the cobrowse configuration, the voiceXML application 2428transmits a prompt that instructs the user to access the web application2430, initiate a cobrowse session, and provide the IVR device 2404 withthe session ID of the cobrowse session. The voiceXML browser 2426receives the prompt and generates an audio prompt for the user—forexample, the voiceXML browser 2426 may use the text to speech module2436 to generate the audio prompt.

The user may access the web application 2430 using a computing device2438 via the internet 2440. The computing device 2438 may be a desktopcomputer, a laptop computer, a tablet, a smartphone, or any othercomputing device able to access the web application server 2412. In someimplementations, the user may click on a button within the UI of theloaded web application 2430 to initiate the cobrowse session.

In response to receiving the request to initiate a cobrowse session, theweb application 2430 uses a web server cobrowse client 2442 tocommunicate with the cobrowse server 2416 to initiate a cobrowse session2444. Upon initiating the cobrowse session 2444, the cobrowse servertransmits the session ID of the cobrowse session 2444 to the web servercobrowse client 2442. The web application 2430 displays the session IDin the UI of the web application 2430. Upon receiving the session ID ofthe cobrowse session 2444, the web server cobrowse client 2442 sends theDOM of the current state of the UI of the web application 2430 to thecobrowse session 2444. Each time the state of the UI of the webapplication 2430 changes, the web server cobrowse client 2442 sends anew DOM to the cobrowse session 2444.

The user may provide the session ID to the IVR device 2404. The voiceXMLbrowser 2426 translates the session ID and provides it to the voiceXMLapplication 2428. The voiceXML application 2428 provides the translatedsession ID to a headless browser 2446. The headless browser 2446 uses anIVR cobrowse client 2448 to transmit the translated session ID to thecobrowse session 2444. The cobrowse server 2416, the web server cobrowseclient 2442, the cobrowse session 2444, and the IVR cobrowse client 2448form a cobrowse platform. In some implementations, the cobrowse platformmay be provided by a remote service, which may be operated by a thirdparty.

In response to receiving the session ID, the cobrowse session 2444transmits the DOM of the current state of the UI of the web application2430 to the IVR cobrowse client 2448, which passes the DOM the headlessbrowser 2446. The headless browser 2446 provides the DOM to the voiceXMLapplication 2428. The voiceXML application 2428 may use the headlessbrowser 2446 and the IVR cobrowse client 2448 to obtain the current DOMof the cobrowse session 2444. In this way, the voiceXML application isable to determine the current state of the UI of the web application2430.

The voiceXML application 2428 may use the cobrowse configuration module2432 to provided prompts and reinforcements to the user based on thegoal associated with the user's request and the current state of the UI.For example, the cobrowse configuration module 2432 includes informationregarding the goal, such as a shortest path to the goal, prompts foreach step of the shortest path, and any reinforcements that areassociated with the shortest path. In some implementations, the cobrowseconfiguration module 2432 may include information generated by the UItesting and enhancement device 110 of the system 100. In otherimplementations, the cobrowse configuration module 2432 may includeinformation generated by another device and/or system.

FIG. 25 is a sequence diagram depicting example requests and associatedresponses between the user, the IVR device 2404, the voiceXMLapplication 2428, the cobrowse configuration module 2432, the headlessbrowser 2446, the IVR cobrowse client 2448, the cobrowse server 2416,and the web application server 2412 during an example initiation of acobrowse session.

The sequence diagram begins with the user providing an audio request fora transaction 2510 to the IVR device 2404. The IVR device 2404translates the request using the speech recognition module 2424 andtransmits the translated request 2514 to the voiceXML application 2428.In response to receiving the translated request, the voiceXMLapplication 2428 requests a cobrowse configuration 2516 associated withthe translated request from the cobrowse configuration module 2432. Thecobrowse configuration module 2432 responds by providing a cobrowseconfiguration 2520 to the voiceXML application 2428.

In response to receiving the cobrowse configuration, the voiceXMLapplication 2428 transmits a cobrowse prompt 2524 to the IVR device2404. The IVR device 2404 then provides the user with a cobrowse prompt2528 that instructs the user to start a cobrowse session using the webapplication 2430 and provide the IVR device 2404 with the session ID ofthe cobrowse session. For example, the IVR device 2404 may use the textto speech module 2436 to generate an audio prompt that is presented tothe user through the telephone 2420. The IVR device 2404 then waits 2532for a response from the user—for example, the user providing a sessionID of the cobrowse session.

Upon hearing the instructions to start a cobrowse session, the usersends a request to start a cobrowse session 2536 to the web applicationserver 2412. In response to receiving the request to start a cobrowsesession, the web application server 2412 transmits a request for acobrowse session 2540 to the cobrowse server 2416. In response toreceiving the cobrowse session request, the cobrowse server 2416 beginsa cobrowse session and returns the session ID of the cobrowse session2544 to the web application server 2412. In response to receiving thesession ID, the web application server 2412 provides the session ID 2548to the user. For example, the web application server 2412 displays thesession ID in the UI of the web application 2430.

The user then provides the session ID 2552 to the IVR device 2404. Inresponse to receiving the session ID from the user, the IVR device 2404translates the session ID and transmits the translated session ID 2556to the voiceXML application 2428. The voiceXML application 2428transmits a request to connect to the cobrowse session 2560 to theheadless browser 2446. The request to connect to the cobrowse sessionincludes the translated session ID. The headless browser 2446 sends therequest to connect to the cobrowse session 2564 to the IVR cobrowseclient 2448. In response to receiving the request to connect, the IVRcobrowse client 2448 transmits a request 2568 to connect to the cobrowsesession to the cobrowse server 2416.

In response to receiving the request to connect, the cobrowse server2416 retrieves the current DOM from the cobrowse session 2444 andtransmits the current DOM 2572 to the IVR cobrowse client 2448, whichthen provides the current DOM 2576 to the headless browser 2446. Theheadless browser 2446 then provides the current DOM 2580 to the voiceXMLapplication 2428. The voiceXML application 2428 uses the current DOM tomonitor the user's progress with respect to reaching the goal in the UI.

FIG. 26 is an example cobrowse sequence diagram depicting examplerequests and associated responses between the user, the IVR device 2404,the voiceXML application 2428, the cobrowse configuration module 2432,the headless browser 2446, the IVR cobrowse client 2448, the cobrowseserver 2416, and the web application server 2412 during a portion of anexample cobrowse session. During the portion of the cobrowse sessiondescribed in the cobrowse sequence diagram 2600, a prompt is presentedto the user regarding an action associated with a step in the shortestpath to a goal in the UI, and the voiceXML application 2428 monitors theuser's progress and provides a reinforcement to aid the user incompleting the action associated with the step.

The cobrowse sequence diagram begins with the voiceXML application 2428requesting configuration data 2603 for the current state of the UI fromthe cobrowse configuration module 2432. In response, the cobrowseconfiguration module 2432 provides the voiceXML application 2428 with aprompt 2606 associated with the current step of the shortest pathassociated with the current state of the UI—for example, “Please enteryour user name.” The voiceXML application 2428 then provides the prompt2609 to the IVR device 2404. The IVR device 2404 presents the prompt2612 to the user. For example, the IVR device 2404 may use the text tospeech module 2436 to generate an audio prompt that is presented to theuser through the telephone 2420.

After the voiceXML application 2428 provides the prompt 2609 to the IVRdevice 2404, the voiceXML application 2428 delays for a predeterminedperiod and then begins to monitor the DOM 2615 of the cobrowse session2444. Specifically, the voiceXML application 2428 requests the DOM 2618from the headless browser 2446. Upon receiving the request for the DOM2618, the headless browser 2446 requests the DOM 2621 from the IVRcobrowse client 2448, which causes the IVR cobrowse client 2448 totransmit a request 2624 to the cobrowse server 2416 for the DOM. Inresponse to receiving the request 2624 for the DOM from the IVR cobrowseclient 2448, the cobrowse server 2416 transmits the current DOM 2627 ofthe cobrowse session 2444 to the IVR cobrowse client 2448. The IVRcobrowse client 2448 provides the current DOM 2630 to the headlessbrowser 2446, which in turn provides the current DOM 2633 to thevoiceXML application 2428. In this way, the voiceXML application 2428 isable to determine if the user has completed the prompted action.

In response to determining that the DOM does not indicate that the userhas performed the prompted action, the voiceXML application 2428requests a reinforcement 2636 associated with the current step from thecobrowse configuration module 2432. In response to receiving the requestfor the reinforcement 2636, the cobrowse configuration module 2432provides a reinforcement 2639 associated with the current step to thevoiceXML application 2428. For example, the reinforcement may be aprompt that indicates the location of a data entry field thatcorresponds to the prompted action.

In response to receiving the reinforcement 2639, the voiceXMLapplication 2428 determines the relative location 2642 of thecoordinates included in the reinforcement. The voiceXML application 2428then provides the reinforcement 2645 to the IVR device 2404 whichpresents the reinforcement 2648 to the user. For example, the IVR device2404 may use the text to speech module 2436 to generate an audioreinforcement.

Upon hearing the audio reinforcement, the user may perform the initiallyprompted action 2651. For example, the user may enter a username in theappropriate text entry field of the UI. In response to the user enteringtext in a text data entry field, the state of the UI changes and the webapplication server 2412 provides an updated DOM 2654 to the cobrowseserver 2416.

After providing the reinforcement 2645 to the IVR device 2404, thevoiceXML application 2428 delays for a predetermined period and thenbegins monitoring the DOM 2657 by requesting the DOM 2660 from theheadless browser 2446 which requests the DOM 2663 from the IVR cobrowseclient 2448. In response to receiving the request for the DOM 2663, theIVR cobrowse client 2448 transmits a request for the DOM 2666 to thecobrowse server 2416. In response to receiving the request for the DOM2666, the cobrowse server 2416 transmits the current DOM 2669 to the IVRcobrowse client 2448. Upon receiving the current DOM 2669, the IVRcobrowse client 2448 provides the current DOM 2672 to the headlessbrowser 2446, which in turn provides the current DOM 2675 to thevoiceXML application 2428. In response to receiving the current DOM2675, the voiceXML application 2428 determines if the initially promptedaction has been completed. If so, the voiceXML application 2428 advancesto the next step 2678 in the shortest path. The customer migrationsystem 2400 continues in this manner, until the voiceXML application2428 determines that the user has reached the goal.

FIGS. 27A-27D are depictions of an example cobrowse session of thecustomer migration system 2400. In FIG. 27A, the output of a screen ofthe computing device 2438 displaying a first state 2702 of the UI of theweb application 2430 is shown. For example, the first state 2702 is theinitial state of the UI of the web application 2430. The IVR device 2404generates a first audio prompt 2704 through the telephone 2420 thatinstructs the user to click on a cobrowse link 2705 located in the firststate 2702 to initiate a cobrowse session. In response to the userclicking on the cobrowse link 2705, the web application server 2412initiates the cobrowse session 2444 on the cobrowse server 2416. Inresponse to initiating the cobrowse session 2444, the cobrowse server2416 provides the web application server 2412 with the session ID of thecobrowse session 2444.

In FIG. 27B, in response to receiving the session ID of the cobrowsesession 2444, the web application 2430 changes the state of the UI suchthat the output of the screen of the computing device 2438 displays asecond state 2706 of the UI of the web application 2430. The secondstate 2706 of the UI includes a window 2710 that contains the session IDof the cobrowse session 2444—(123456). In response to generating thefirst audio prompt 2704, the IVR device 2404 generates a second audioprompt 2708 through the telephone 2420 that requests the user to providethe IVR device 2404 with the session ID of the cobrowse session 2444.

In FIG. 27C, in response to the user clicking an OK button in the window2710, the web application 2430 changes the state of the UI, such thatthe screen of the computing device 2438 displays the first state 2702.In other words, the output of the screen of the computing device nolonger includes the window 2710. In response to changing the state ofthe UI, the web application server 2412 provides a DOM of the firststate 2702 to the cobrowse server 2416. In response to the userproviding the IVR device 2404 with the session ID of the cobrowsesession 2444, the IVR device 2404 provides the IVR server 2408 with thesession ID.

The IVR server 2408 then connects to the cobrowse session 2444 andreceives the current DOM of the cobrowse session—or example, the DOM ofthe first state 2702. The IVR server 2408, based on the DOM of the firststate 2702 and prompt data from the cobrowse configuration module 2432,causes the IVR device 2404 to generate a third audio prompt 2714. Thethird audio prompt 2714 instructs the user to enter their username intothe web application.

In FIG. 27D, in response to determining that the user has not completedthe action associated with the third audio prompt 2714, the IVR server2408, based on the DOM of the first state 2702 and reinforcementinformation from the cobrowse configuration module 2432, causes the IVRdevice 2404 to generate an audio reinforcement 2716. The audioreinforcement 2716 instructs the user to perform the original action andprovides the user with additional information relevant to completing theaction. For example, the audio reinforcement 2716 provides the user withthe location of the data entry field to be used to enter their username.In this way, the IVR server 2408 is able to guide the user through theweb application 2430 to a specific goal.

High-Volume Pharmacy

FIG. 28 is a block diagram of an example implementation of a system 2800for a high-volume pharmacy. While the system 2800 is generally describedas being deployed in a high-volume pharmacy or a fulfillment center (forexample, a mail order pharmacy, a direct delivery pharmacy, etc.), thesystem 2800 and/or components of the system 2800 may otherwise bedeployed (for example, in a lower-volume pharmacy, etc.). A high-volumepharmacy may be a pharmacy that is capable of filling at least someprescriptions mechanically. The system 2800 may include a benefitmanager device 2802 and a pharmacy device 2806 in communication witheach other directly and/or over a network 2804.

The system 2800 may also include one or more user device(s) 2808. Auser, such as a pharmacist, patient, data analyst, health planadministrator, etc., may access the benefit manager device 2802 or thepharmacy device 2806 using the user device 2808. The user device 2808may be a desktop computer, a laptop computer, a tablet, a smartphone,etc.

The benefit manager device 2802 is a device operated by an entity thatis at least partially responsible for creation and/or management of thepharmacy or drug benefit. While the entity operating the benefit managerdevice 2802 is typically a pharmacy benefit manager (PBM), otherentities may operate the benefit manager device 2802 on behalf ofthemselves or other entities (such as PBMs). For example, the benefitmanager device 2802 may be operated by a health plan, a retail pharmacychain, a drug wholesaler, a data analytics or other type ofsoftware-related company, etc. In some implementations, a PBM thatprovides the pharmacy benefit may provide one or more additionalbenefits including a medical or health benefit, a dental benefit, avision benefit, a wellness benefit, a radiology benefit, a pet carebenefit, an insurance benefit, a long term care benefit, a nursing homebenefit, etc. The PBM may, in addition to its PBM operations, operateone or more pharmacies. The pharmacies may be retail pharmacies, mailorder pharmacies, etc.

Some of the operations of the PBM that operates the benefit managerdevice 2802 may include the following activities and processes. A member(or a person on behalf of the member) of a pharmacy benefit plan mayobtain a prescription drug at a retail pharmacy location (e.g., alocation of a physical store) from a pharmacist or a pharmacisttechnician. The member may also obtain the prescription drug throughmail order drug delivery from a mail order pharmacy location, such asthe system 2800. In some implementations, the member may obtain theprescription drug directly or indirectly through the use of a machine,such as a kiosk, a vending unit, a mobile electronic device, or adifferent type of mechanical device, electrical device, electroniccommunication device, and/or computing device. Such a machine may befilled with the prescription drug in prescription packaging, which mayinclude multiple prescription components, by the system 2800. Thepharmacy benefit plan is administered by or through the benefit managerdevice 2802.

The member may have a copayment for the prescription drug that reflectsan amount of money that the member is responsible to pay the pharmacyfor the prescription drug. The money paid by the member to the pharmacymay come from, as examples, personal funds of the member, a healthsavings account (HSA) of the member or the member's family, a healthreimbursement arrangement (HRA) of the member or the member's family, ora flexible spending account (FSA) of the member or the member's family.In some instances, an employer of the member may directly or indirectlyfund or reimburse the member for the copayments.

The amount of the copayment required by the member may vary acrossdifferent pharmacy benefit plans having different plan sponsors orclients and/or for different prescription drugs. The member's copaymentmay be a flat copayment (in one example, $10), coinsurance (in oneexample, 10%), and/or a deductible (for example, responsibility for thefirst $500 of annual prescription drug expense, etc.) for certainprescription drugs, certain types and/or classes of prescription drugs,and/or all prescription drugs. The copayment may be stored in a storagedevice 2810 or determined by the benefit manager device 2802.

In some instances, the member may not pay the copayment or may only paya portion of the copayment for the prescription drug. For example, if ausual and customary cost for a generic version of a prescription drug is$4, and the member's flat copayment is $20 for the prescription drug,the member may only need to pay $4 to receive the prescription drug. Inanother example involving a worker's compensation claim, no copaymentmay be due by the member for the prescription drug.

In addition, copayments may also vary based on different deliverychannels for the prescription drug. For example, the copayment forreceiving the prescription drug from a mail order pharmacy location maybe less than the copayment for receiving the prescription drug from aretail pharmacy location.

In conjunction with receiving a copayment (if any) from the member anddispensing the prescription drug to the member, the pharmacy submits aclaim to the PBM for the prescription drug. After receiving the claim,the PBM (such as by using the benefit manager device 2802) may performcertain adjudication operations including verifying eligibility for themember, identifying/reviewing an applicable formulary for the member todetermine any appropriate copayment, coinsurance, and deductible for theprescription drug, and performing a drug utilization review (DUR) forthe member. Further, the PBM may provide a response to the pharmacy (forexample, the pharmacy system 2800) following performance of at leastsome of the aforementioned operations.

As part of the adjudication, a plan sponsor (or the PBM on behalf of theplan sponsor) ultimately reimburses the pharmacy for filling theprescription drug when the prescription drug was successfullyadjudicated. The aforementioned adjudication operations generally occurbefore the copayment is received and the prescription drug is dispensed.However in some instances, these operations may occur simultaneously,substantially simultaneously, or in a different order. In addition, moreor fewer adjudication operations may be performed as at least part ofthe adjudication process.

The amount of reimbursement paid to the pharmacy by a plan sponsorand/or money paid by the member may be determined at least partiallybased on types of pharmacy networks in which the pharmacy is included.In some implementations, the amount may also be determined based onother factors. For example, if the member pays the pharmacy for theprescription drug without using the prescription or drug benefitprovided by the PBM, the amount of money paid by the member may behigher than when the member uses the prescription or drug benefit. Insome implementations, the amount of money received by the pharmacy fordispensing the prescription drug and for the prescription drug itselfmay be higher than when the member uses the prescription or drugbenefit. Some or all of the foregoing operations may be performed byexecuting instructions stored in the benefit manager device 2802 and/oran additional device.

Examples of the network 2804 include a Global System for MobileCommunications (GSM) network, a code division multiple access (CDMA)network, 3rd Generation Partnership Project (3GPP), an Internet Protocol(IP) network, a Wireless Application Protocol (WAP) network, or an IEEE802.11 standards network, as well as various combinations of the abovenetworks. The network 2804 may include an optical network. The network2804 may be a local area network or a global communication network, suchas the Internet. In some implementations, the network 2804 may include anetwork dedicated to prescription orders: a prescribing network such asthe electronic prescribing network operated by Surescripts of Arlington,Va.

Moreover, although the system shows a single network 2804, multiplenetworks can be used. The multiple networks may communicate in seriesand/or parallel with each other to link the devices 2802-2810.

The pharmacy device 2806 may be a device associated with a retailpharmacy location (e.g., an exclusive pharmacy location, a grocery storewith a retail pharmacy, or a general sales store with a retail pharmacy)or other type of pharmacy location at which a member attempts to obtaina prescription. The pharmacy may use the pharmacy device 2806 to submitthe claim to the PBM for adjudication.

Additionally, in some implementations, the pharmacy device 2806 mayenable information exchange between the pharmacy and the PBM. Forexample, this may allow the sharing of member information such as drughistory that may allow the pharmacy to better service a member (forexample, by providing more informed therapy consultation and druginteraction information). In some implementations, the benefit managerdevice 2802 may track prescription drug fulfillment and/or otherinformation for users that are not members, or have not identifiedthemselves as members, at the time (or in conjunction with the time) inwhich they seek to have a prescription filled at a pharmacy.

The pharmacy device 2806 may include a pharmacy fulfillment device 2812,an order processing device 2814, and a pharmacy management device 2816in communication with each other directly and/or over the network 2804.The order processing device 2814 may receive information regardingfilling prescriptions and may direct an order component to one or moredevices of the pharmacy fulfillment device 2812 at a pharmacy. Thepharmacy fulfillment device 2812 may fulfill, dispense, aggregate,and/or pack the order components of the prescription drugs in accordancewith one or more prescription orders directed by the order processingdevice 2814.

In general, the order processing device 2814 is a device located withinor otherwise associated with the pharmacy to enable the pharmacyfulfilment device 2812 to fulfill a prescription and dispenseprescription drugs. In some implementations, the order processing device2814 may be an external order processing device separate from thepharmacy and in communication with other devices located within thepharmacy.

For example, the external order processing device may communicate withan internal pharmacy order processing device and/or other deviceslocated within the system 2800. In some implementations, the externalorder processing device may have limited functionality (e.g., asoperated by a user requesting fulfillment of a prescription drug), whilethe internal pharmacy order processing device may have greaterfunctionality (e.g., as operated by a pharmacist).

The order processing device 2814 may track the prescription order as itis fulfilled by the pharmacy fulfillment device 2812. The prescriptionorder may include one or more prescription drugs to be filled by thepharmacy. The order processing device 2814 may make pharmacy routingdecisions and/or order consolidation decisions for the particularprescription order. The pharmacy routing decisions include whatdevice(s) in the pharmacy are responsible for filling or otherwisehandling certain portions of the prescription order. The orderconsolidation decisions include whether portions of one prescriptionorder or multiple prescription orders should be shipped together for auser or a user family. The order processing device 2814 may also trackand/or schedule literature or paperwork associated with eachprescription order or multiple prescription orders that are beingshipped together. In some implementations, the order processing device2814 may operate in combination with the pharmacy management device2816.

The order processing device 2814 may include circuitry, a processor, amemory to store data and instructions, and communication functionality.The order processing device 2814 is dedicated to performing processes,methods, and/or instructions described in this application. Other typesof electronic devices may also be used that are specifically configuredto implement the processes, methods, and/or instructions described infurther detail below.

In some implementations, at least some functionality of the orderprocessing device 2814 may be included in the pharmacy management device2816. The order processing device 2814 may be in a client-serverrelationship with the pharmacy management device 2816, in a peer-to-peerrelationship with the pharmacy management device 2816, or in a differenttype of relationship with the pharmacy management device 2816. The orderprocessing device 2814 and/or the pharmacy management device 2816 maycommunicate directly (for example, such as by using a local storage)and/or through the network 2804 (such as by using a cloud storageconfiguration, software as a service, etc.) with the storage device2810.

The storage device 2810 may include: non-transitory storage (forexample, memory, hard disk, CD-ROM, etc.) in communication with thebenefit manager device 2802 and/or the pharmacy device 2806 directlyand/or over the network 2804. The non-transitory storage may store orderdata 2818, member data 2820, claims data 2822, drug data 2824,prescription data 2826, and/or plan sponsor data 2828. Further, thesystem 2800 may include additional devices, which may communicate witheach other directly or over the network 2804.

The order data 2818 may be related to a prescription order. The orderdata may include type of the prescription drug (for example, drug nameand strength) and quantity of the prescription drug. The order data 2818may also include data used for completion of the prescription, such asprescription materials. In general, prescription materials include anelectronic copy of information regarding the prescription drug forinclusion with or otherwise in conjunction with the fulfilledprescription. The prescription materials may include electronicinformation regarding drug interaction warnings, recommended usage,possible side effects, expiration date, date of prescribing, etc. Theorder data 2818 may be used by a high-volume fulfillment center tofulfill a pharmacy order.

In some implementations, the order data 2818 includes verificationinformation associated with fulfillment of the prescription in thepharmacy. For example, the order data 2818 may include videos and/orimages taken of (i) the prescription drug prior to dispensing, duringdispensing, and/or after dispensing, (ii) the prescription container(for example, a prescription container and sealing lid, prescriptionpackaging, etc.) used to contain the prescription drug prior todispensing, during dispensing, and/or after dispensing, (iii) thepackaging and/or packaging materials used to ship or otherwise deliverthe prescription drug prior to dispensing, during dispensing, and/orafter dispensing, and/or (iv) the fulfillment process within thepharmacy. Other types of verification information such as barcode dataread from pallets, bins, trays, or carts used to transport prescriptionswithin the pharmacy may also be stored as order data 2818.

The member data 2820 includes information regarding the membersassociated with the PBM. The information stored as member data 2820 mayinclude personal information, personal health information, protectedhealth information, etc. Examples of the member data 2820 include name,address, telephone number, e-mail address, prescription drug history,etc. The member data 2820 may include a plan sponsor identifier thatidentifies the plan sponsor associated with the member and/or a memberidentifier that identifies the member to the plan sponsor. The memberdata 2820 may include a member identifier that identifies the plansponsor associated with the user and/or a user identifier thatidentifies the user to the plan sponsor. The member data 2820 may alsoinclude dispensation preferences such as type of label, type of cap,message preferences, language preferences, etc.

The member data 2820 may be accessed by various devices in the pharmacy(for example, the high-volume fulfillment center, etc.) to obtaininformation used for fulfillment and shipping of prescription orders. Insome implementations, an external order processing device operated by oron behalf of a member may have access to at least a portion of themember data 2820 for review, verification, or other purposes.

In some implementations, the member data 2820 may include informationfor persons who are users of the pharmacy but are not members in thepharmacy benefit plan being provided by the PBM. For example, theseusers may obtain drugs directly from the pharmacy, through a privatelabel service offered by the pharmacy, the high-volume fulfillmentcenter, or otherwise. In general, the use of the terms “member” and“user” may be used interchangeably.

The claims data 2822 includes information regarding pharmacy claimsadjudicated by the PBM under a drug benefit program provided by the PBMfor one or more plan sponsors. In general, the claims data 2822 includesan identification of the client that sponsors the drug benefit programunder which the claim is made, and/or the member that purchased theprescription drug giving rise to the claim, the prescription drug thatwas filled by the pharmacy (e.g., the national drug code number, etc.),the dispensing date, generic indicator, generic product identifier (GPI)number, medication class, the cost of the prescription drug providedunder the drug benefit program, the copayment/coinsurance amount, rebateinformation, and/or member eligibility, etc. Additional information maybe included.

In some implementations, other types of claims beyond prescription drugclaims may be stored in the claims data 2822. For example, medicalclaims, dental claims, wellness claims, or other types ofhealth-care-related claims for members may be stored as a portion of theclaims data 2822.

In some implementations, the claims data 2822 includes claims thatidentify the members with whom the claims are associated. Additionallyor alternatively, the claims data 2822 may include claims that have beende-identified (that is, associated with a unique identifier but not witha particular, identifiable member).

The drug data 2824 may include drug name (e.g., technical name and/orcommon name), other names by which the drug is known, activeingredients, an image of the drug (such as in pill form), etc. The drugdata 2824 may include information associated with a single medication ormultiple medications.

The prescription data 2826 may include information regardingprescriptions that may be issued by prescribers on behalf of users, whomay be members of the pharmacy benefit plan—for example, to be filled bya pharmacy. Examples of the prescription data 2826 include user names,medication or treatment (such as lab tests), dosing information, etc.The prescriptions may include electronic prescriptions or paperprescriptions that have been scanned. In some implementations, thedosing information reflects a frequency of use (e.g., once a day, twicea day, before each meal, etc.) and a duration of use (e.g., a few days,a week, a few weeks, a month, etc.).

In some implementations, the order data 2818 may be linked to associatedmember data 2820, claims data 2822, drug data 2824, and/or prescriptiondata 2826.

The plan sponsor data 2828 includes information regarding the plansponsors of the PBM. Examples of the plan sponsor data 2828 includecompany name, company address, contact name, contact telephone number,contact e-mail address, etc.

FIG. 29 illustrates the pharmacy fulfillment device 2812 according to anexample implementation. The pharmacy fulfillment device 2812 may be usedto process and fulfill prescriptions and prescription orders. Afterfulfillment, the fulfilled prescriptions are packed for shipping.

The pharmacy fulfillment device 2812 may include devices incommunication with the benefit manager device 2802, the order processingdevice 2814, and/or the storage device 2810, directly or over thenetwork 2804. Specifically, the pharmacy fulfillment device 2812 mayinclude pallet sizing and pucking device(s) 2906, loading device(s)2908, inspect device(s) 2910, unit of use device(s) 2912, automateddispensing device(s) 2914, manual fulfillment device(s) 2916, reviewdevices 2918, imaging device(s) 2920, cap device(s) 2922, accumulationdevices 2924, packing device(s) 2926, literature device(s) 2928, unit ofuse packing device(s) 2930, and mail manifest device(s) 2932. Further,the pharmacy fulfillment device 2812 may include additional devices,which may communicate with each other directly or over the network 2804.

In some implementations, operations performed by one of these devices2906-2932 may be performed sequentially, or in parallel with theoperations of another device as may be coordinated by the orderprocessing device 2814. In some implementations, the order processingdevice 2814 tracks a prescription with the pharmacy based on operationsperformed by one or more of the devices 2906-2932.

In some implementations, the pharmacy fulfillment device 2812 maytransport prescription drug containers, for example, among the devices2906-2932 in the high-volume fulfillment center, by use of pallets. Thepallet sizing and pucking device 2906 may configure pucks in a pallet. Apallet may be a transport structure for a number of prescriptioncontainers, and may include a number of cavities. A puck may be placedin one or more than one of the cavities in a pallet by the pallet sizingand pucking device 2906. The puck may include a receptacle sized andshaped to receive a prescription container. Such containers may besupported by the pucks during carriage in the pallet. Different pucksmay have differently sized and shaped receptacles to accommodatecontainers of differing sizes, as may be appropriate for differentprescriptions.

The arrangement of pucks in a pallet may be determined by the orderprocessing device 2814 based on prescriptions that the order processingdevice 2814 decides to launch. The arrangement logic may be implementeddirectly in the pallet sizing and pucking device 2906. Once aprescription is set to be launched, a puck suitable for the appropriatesize of container for that prescription may be positioned in a pallet bya robotic arm or pickers. The pallet sizing and pucking device 2906 maylaunch a pallet once pucks have been configured in the pallet.

The loading device 2908 may load prescription containers into the puckson a pallet by a robotic arm, a pick and place mechanism (also referredto as pickers), etc. In various implementations, the loading device 2908has robotic arms or pickers to grasp a prescription container and moveit to and from a pallet or a puck. The loading device 2908 may alsoprint a label that is appropriate for a container that is to be loadedonto the pallet, and apply the label to the container. The pallet may belocated on a conveyor assembly during these operations (e.g., at thehigh-volume fulfillment center, etc.).

The inspect device 2910 may verify that containers in a pallet arecorrectly labeled and in the correct spot on the pallet. The inspectdevice 2910 may scan the label on one or more containers on the pallet.Labels of containers may be scanned or imaged in full or in part by theinspect device 2910. Such imaging may occur after the container has beenlifted out of its puck by a robotic arm, picker, etc., or may beotherwise scanned or imaged while retained in the puck. In someimplementations, images and/or video captured by the inspect device 2910may be stored in the storage device 2810 as order data 2818.

The unit of use device 2912 may temporarily store, monitor, label,and/or dispense unit of use products. In general, unit of use productsare prescription drug products that may be delivered to a user or memberwithout being repackaged at the pharmacy. These products may includepills in a container, pills in a blister pack, inhalers, etc.Prescription drug products dispensed by the unit of use device 2912 maybe packaged individually or collectively for shipping, or may be shippedin combination with other prescription drugs dispensed by other devicesin the high-volume fulfillment center.

At least some of the operations of the devices 2906-2932 may be directedby the order processing device 2814. For example, the manual fulfillmentdevice 2916, the review device 2918, the automated dispensing device2914, and/or the packing device 2926, etc. may receive instructionsprovided by the order processing device 2814.

The automated dispensing device 2914 may include one or more devicesthat dispense prescription drugs or pharmaceuticals into prescriptioncontainers in accordance with one or multiple prescription orders. Ingeneral, the automated dispensing device 2914 may include mechanical andelectronic components with, in some implementations, software and/orlogic to facilitate pharmaceutical dispensing that would otherwise beperformed in a manual fashion by a pharmacist and/or pharmacisttechnician. For example, the automated dispensing device 2914 mayinclude high-volume fillers that fill a number of prescription drugtypes at a rapid rate and blister pack machines that dispense and packdrugs into a blister pack. Prescription drugs dispensed by the automateddispensing devices 2914 may be packaged individually or collectively forshipping, or may be shipped in combination with other prescription drugsdispensed by other devices in the high-volume fulfillment center.

The manual fulfillment device 2916 controls how prescriptions aremanually fulfilled. For example, the manual fulfillment device 2916 mayreceive or obtain a container and enable fulfillment of the container bya pharmacist or pharmacy technician. In some implementations, the manualfulfillment device 2916 provides the filled container to another devicein the pharmacy fulfillment devices 2812 to be joined with othercontainers in a prescription order for a user or member.

In general, manual fulfillment may include operations at least partiallyperformed by a pharmacist or a pharmacy technician. For example, aperson may retrieve a supply of the prescribed drug, may make anobservation, may count out a prescribed quantity of drugs and place theminto a prescription container, etc. Some portions of the manualfulfillment process may be automated by use of a machine. For example,counting of capsules, tablets, or pills may be at least partiallyautomated (such as through use of a pill counter). Prescription drugsdispensed by the manual fulfillment device 2916 may be packagedindividually or collectively for shipping, or may be shipped incombination with other prescription drugs dispensed by other devices inthe high-volume fulfillment center.

The review device 2918 may process prescription containers to bereviewed by a pharmacist for proper pill count, exception handling,prescription verification, etc. Fulfilled prescriptions may be manuallyreviewed and/or verified by a pharmacist, as may be required by state orlocal law. A pharmacist or other licensed pharmacy person who maydispense certain drugs in compliance with local and/or other laws mayoperate the review device 2918 and visually inspect a prescriptioncontainer that has been filled with a prescription drug. The pharmacistmay review, verify, and/or evaluate drug quantity, drug strength, and/ordrug interaction concerns, or otherwise perform pharmacist services. Thepharmacist may also handle containers which have been flagged as anexception, such as containers with unreadable labels, containers forwhich the associated prescription order has been canceled, containerswith defects, etc. In an example, the manual review can be performed ata manual review station.

The imaging device 2920 may image containers once they have been filledwith pharmaceuticals. The imaging device 2920 may measure a fill heightof the pharmaceuticals in the container based on the obtained image todetermine if the container is filled to the correct height given thetype of pharmaceutical and the number of pills in the prescription.Images of the pills in the container may also be obtained to detect thesize of the pills themselves and markings thereon. The images may betransmitted to the order processing device 2814 and/or stored in thestorage device 2810 as part of the order data 2818.

The cap device 2922 may be used to cap or otherwise seal a prescriptioncontainer. In some implementations, the cap device 2922 may secure aprescription container with a type of cap in accordance with a userpreference (e.g., a preference regarding child resistance, etc.), a plansponsor preference, a prescriber preference, etc. The cap device 2922may also etch a message into the cap, although this process may beperformed by a subsequent device in the high-volume fulfillment center.

The accumulation device 2924 accumulates various containers ofprescription drugs in a prescription order. The accumulation device 2924may accumulate prescription containers from various devices or areas ofthe pharmacy. For example, the accumulation device 2924 may accumulateprescription containers from the unit of use device 2912, the automateddispensing device 2914, the manual fulfillment device 2916, and thereview device 2918. The accumulation device 2924 may be used to groupthe prescription containers prior to shipment to the member.

The literature device 2928 prints, or otherwise generates, literature toinclude with each prescription drug order. The literature may be printedon multiple sheets of substrates, such as paper, coated paper, printablepolymers, or combinations of the above substrates. The literatureprinted by the literature device 2928 may include information requiredto accompany the prescription drugs included in a prescription order,other information related to prescription drugs in the order, financialinformation associated with the order (for example, an invoice or anaccount statement), etc.

In some implementations, the literature device 2928 folds or otherwiseprepares the literature for inclusion with a prescription drug order(e.g., in a shipping container). In other implementations, theliterature device 2928 prints the literature and is separate fromanother device that prepares the printed literature for inclusion with aprescription order.

The packing device 2926 packages the prescription order in preparationfor shipping the order. The packing device 2926 may box, bag, orotherwise package the fulfilled prescription order for delivery. Thepacking device 2926 may further place inserts (e.g., literature or otherpapers, etc.) into the packaging received from the literature device2928. For example, bulk prescription orders may be shipped in a box,while other prescription orders may be shipped in a bag, which may be awrap seal bag.

The packing device 2926 may label the box or bag with an address and arecipient's name. The label may be printed and affixed to the bag orbox, be printed directly onto the bag or box, or otherwise associatedwith the bag or box. The packing device 2926 may sort the box or bag formailing in an efficient manner (e.g., sort by delivery address, etc.).The packing device 2926 may include ice or temperature sensitiveelements for prescriptions that are to be kept within a temperaturerange during shipping (for example, this may be necessary in order toretain efficacy). The ultimate package may then be shipped throughpostal mail, through a mail order delivery service that ships via groundand/or air (e.g., UPS, FEDEX, or DHL, etc.), through a delivery service,through a locker box at a shipping site (e.g., AMAZON locker or a POBox, etc.), or otherwise.

The unit of use packing device 2930 packages a unit of use prescriptionorder in preparation for shipping the order. The unit of use packingdevice 2930 may include manual scanning of containers to be bagged forshipping to verify each container in the order. In an exampleimplementation, the manual scanning may be performed at a manualscanning station. The pharmacy fulfillment device 2812 may also includea mail manifest device 2932 to print mailing labels used by the packingdevice 2926 and may print shipping manifests and packing lists.

While the pharmacy fulfillment device 2812 in FIG. 29 is shown toinclude single devices 2906-2932, multiple devices may be used. Whenmultiple devices are present, the multiple devices may be of the samedevice type or models, or may be a different device type or model. Thetypes of devices 2906-2932 shown in FIG. 29 are example devices. Inother configurations of the system 2800, lesser, additional, ordifferent types of devices may be included.

Moreover, multiple devices may share processing and/or memory resources.The devices 2906-2932 may be located in the same area or in differentlocations. For example, the devices 2906-2932 may be located in abuilding or set of adjoining buildings. The devices 2906-2932 may beinterconnected (such as by conveyors), networked, and/or otherwise incontact with one another or integrated with one another (e.g., at thehigh-volume fulfillment center, etc.). In addition, the functionality ofa device may be split among a number of discrete devices and/or combinedwith other devices.

FIG. 30 illustrates the order processing device 2814 according to anexample implementation. The order processing device 2814 may be used byone or more operators to generate prescription orders, make routingdecisions, make prescription order consolidation decisions, trackliterature with the system 2800, and/or view order status and otherorder related information. For example, the prescription order may becomprised of order components.

The order processing device 2814 may receive instructions to fulfill anorder without operator intervention. An order component may include aprescription drug fulfilled by use of a container through the system2800. The order processing device 2814 may include an order verificationsubsystem 3002, an order control subsystem 3004, and/or an ordertracking subsystem 3006. Other subsystems may also be included in theorder processing device 2814.

The order verification subsystem 3002 may communicate with the benefitmanager device 2802 to verify the eligibility of the member and reviewthe formulary to determine appropriate copayment, coinsurance, anddeductible for the prescription drug and/or perform a DUR (drugutilization review). Other communications between the order verificationsubsystem 3002 and the benefit manager device 2802 may be performed fora variety of purposes.

The order control subsystem 3004 controls various movements of thecontainers and/or pallets along with various filling functions duringtheir progression through the system 2800. In some implementations, theorder control subsystem 3004 may identify the prescribed drug in one ormore than one prescription orders as capable of being fulfilled by theautomated dispensing device 2914. The order control subsystem 3004 maydetermine which prescriptions are to be launched and may determine thata pallet of automated-fill containers is to be launched.

The order control subsystem 3004 may determine that an automated-fillprescription of a specific pharmaceutical is to be launched and mayexamine a queue of orders awaiting fulfillment for other prescriptionorders, which will be filled with the same pharmaceutical. The ordercontrol subsystem 3004 may then launch orders with similarautomated-fill pharmaceutical needs together in a pallet to theautomated dispensing device 2914. As the devices 2906-2932 may beinterconnected by a system of conveyors or other container movementsystems, the order control subsystem 3004 may control various conveyors:for example, to deliver the pallet from the loading device 2908 to themanual fulfillment device 2916 from the literature device 2928,paperwork as needed to fill the prescription.

The order tracking subsystem 3006 may track a prescription order duringits progress toward fulfillment. The order tracking subsystem 3006 maytrack, record, and/or update order history, order status, etc. The ordertracking subsystem 3006 may store data locally (for example, in amemory) or as a portion of the order data 2818 stored in the storagedevice 2810.

CONCLUSION

The foregoing description is merely illustrative in nature and is in noway intended to limit the disclosure, its application, or uses. Thebroad teachings of the disclosure can be implemented in a variety offorms. Therefore, while this disclosure includes particular examples,the true scope of the disclosure should not be so limited since othermodifications will become apparent upon a study of the drawings, thespecification, and the following claims. It should be understood thatone or more steps within a method may be executed in different order (orconcurrently) without altering the principles of the present disclosure.Further, although each of the embodiments is described above as havingcertain features, any one or more of those features described withrespect to any embodiment of the disclosure can be implemented in and/orcombined with features of any of the other embodiments, even if thatcombination is not explicitly described. In other words, the describedembodiments are not mutually exclusive, and permutations of one or moreembodiments with one another remain within the scope of this disclosure.

Spatial and functional relationships between elements (for example,between modules) are described using various terms, including“connected,” “engaged,” “interfaced,” and “coupled.” Unless explicitlydescribed as being “direct,” when a relationship between first andsecond elements is described in the above disclosure, that relationshipencompasses a direct relationship where no other intervening elementsare present between the first and second elements, and also an indirectrelationship where one or more intervening elements are present (eitherspatially or functionally) between the first and second elements. Asused herein, the phrase at least one of A, B, and C should be construedto mean a logical (A OR B OR C), using a non-exclusive logical OR, andshould not be construed to mean “at least one of A, at least one of B,and at least one of C.”

In the figures, the direction of an arrow, as indicated by thearrowhead, generally demonstrates the flow of information (such as dataor instructions) that is of interest to the illustration. For example,when element A and element B exchange a variety of information butinformation transmitted from element A to element B is relevant to theillustration, the arrow may point from element A to element B. Thisunidirectional arrow does not imply that no other information istransmitted from element B to element A. Further, for information sentfrom element A to element B, element B may send requests for, or receiptacknowledgements of, the information to element A. The term subset doesnot necessarily require a proper subset. In other words, a first subsetof a first set may be coextensive with (equal to) the first set.

In this application, including the definitions below, the term “module”or the term “controller” may be replaced with the term “circuit.” Theterm “module” may refer to, be part of, or include processor hardware(shared, dedicated, or group) that executes code and memory hardware(shared, dedicated, or group) that stores code executed by the processorhardware.

The module may include one or more interface circuits. In some examples,the interface circuit(s) may implement wired or wireless interfaces thatconnect to a local area network (LAN) or a wireless personal areanetwork (WPAN). Examples of a LAN are Institute of Electrical andElectronics Engineers (IEEE) Standard 802.11-2016 (also known as theWIFI wireless networking standard) and IEEE Standard 802.3-2015 (alsoknown as the ETHERNET wired networking standard). Examples of a WPAN arethe BLUETOOTH wireless networking standard from the Bluetooth SpecialInterest Group and IEEE Standard 802.15.4.

The module may communicate with other modules using the interfacecircuit(s). Although the module may be depicted in the presentdisclosure as logically communicating directly with other modules, invarious implementations the module may actually communicate via acommunications system. The communications system includes physicaland/or virtual networking equipment such as hubs, switches, routers, andgateways. In some implementations, the communications system connects toor traverses a wide area network (WAN) such as the Internet. Forexample, the communications system may include multiple LANs connectedto each other over the Internet or point-to-point leased lines usingtechnologies including Multiprotocol Label Switching (MPLS) and virtualprivate networks (VPNs).

In various implementations, the functionality of the module may bedistributed among multiple modules that are connected via thecommunications system. For example, multiple modules may implement thesame functionality distributed by a load balancing system. In a furtherexample, the functionality of the module may be split between a server(also known as remote, or cloud) module and a client (or, user) module.

The term code, as used above, may include software, firmware, and/ormicrocode, and may refer to programs, routines, functions, classes, datastructures, and/or objects. Shared processor hardware encompasses asingle microprocessor that executes some or all code from multiplemodules. Group processor hardware encompasses a microprocessor that, incombination with additional microprocessors, executes some or all codefrom one or more modules. References to multiple microprocessorsencompass multiple microprocessors on discrete dies, multiplemicroprocessors on a single die, multiple cores of a singlemicroprocessor, multiple threads of a single microprocessor, or acombination of the above.

Shared memory hardware encompasses a single memory device that storessome or all code from multiple modules. Group memory hardwareencompasses a memory device that, in combination with other memorydevices, stores some or all code from one or more modules.

The term memory hardware is a subset of the term computer-readablemedium. The term computer-readable medium, as used herein, does notencompass transitory electrical or electromagnetic signals propagatingthrough a medium (such as on a carrier wave); the term computer-readablemedium is therefore considered tangible and non-transitory. Non-limitingexamples of a non-transitory computer-readable medium are nonvolatilememory devices (such as a flash memory device, an erasable programmableread-only memory device, or a mask read-only memory device), volatilememory devices (such as a static random access memory device or adynamic random access memory device), magnetic storage media (such as ananalog or digital magnetic tape or a hard disk drive), and opticalstorage media (such as a CD, a DVD, or a Blu-ray Disc).

The apparatuses and methods described in this application may bepartially or fully implemented by a special purpose computer created byconfiguring a general purpose computer to execute one or more particularfunctions embodied in computer programs. The functional blocks andflowchart elements described above serve as software specifications,which can be translated into the computer programs by the routine workof a skilled technician or programmer.

The computer programs include processor-executable instructions that arestored on at least one non-transitory computer-readable medium. Thecomputer programs may also include or rely on stored data. The computerprograms may encompass a basic input/output system (BIOS) that interactswith hardware of the special purpose computer, device drivers thatinteract with particular devices of the special purpose computer, one ormore operating systems, user applications, background services,background applications, etc.

The computer programs may include: (i) descriptive text to be parsed,such as HTML (hypertext markup language), XML (extensible markuplanguage), or JSON (JavaScript Object Notation), (ii) assembly code,(iii) object code generated from source code by a compiler, (iv) sourcecode for execution by an interpreter, (v) source code for compilationand execution by a just-in-time compiler, etc. As examples only, sourcecode may be written using syntax from languages including C, C++, C #,Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl,Pascal, Curl, OCaml, Javascript®, HTML5 (Hypertext Markup Language 5threvision), Ada, ASP (Active Server Pages), PHP (PHP: HypertextPreprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, VisualBasic®, Lua, MATLAB, SIMULINK, and Python®.

What is claimed is:
 1. A method for integrating a telephone system and a computer system comprising: obtaining a computer-readable command based on user audio input, the computer-readable command being associated with a web application; determining a goal in the web application associated with the computer-readable command; obtaining information indicating a shortest navigation path to the goal in the web application relative to other navigation paths to the goal in the web application; receiving a document object model (DOM) of a current state of the web application from a cobrowse session for a web server hosting the web application; determining a next navigation action along the shortest navigation path; generating a user prompt based on the next navigation action; and receiving an updated DOM in response to user execution of the next navigation action.
 2. The method of claim 1, further comprising: obtaining a user cobrowse session identifier; and transmitting the user cobrowse session identifier to the cobrowse session, wherein receiving the DOM of the current state includes receiving the DOM in response to transmitting the user cobrowse session identifier.
 3. The method of claim 1, wherein obtaining information indicating the shortest navigation path comprises: determining the shortest navigation path among the other navigation paths to the goal based on a number of inputs required metric, a number of mouse movements metric, a number of keystrokes metric, a number of mouse click events metric, or combinations thereof.
 4. The method of claim 1, wherein the user prompt includes a user voice prompt, a user visual prompt, or combinations thereof.
 5. The method of claim 1, wherein the goal includes obtain an order status.
 6. The method of claim 1, wherein the shortest navigation path has the smallest total Euclidean distance of mouse travel relative to the other navigation paths.
 7. The method of claim 1, wherein the current state of the web application reflects a user interface current state from a user perspective.
 8. A system for integrating a telephone system and a computing system, the system comprising: an interactive voice response (IVR) platform configured to: obtain a computer-readable command based on user audio input, and in response to obtaining the computer-readable command, (i) determine a goal in a web application associated with the computer-readable command, and (ii) obtain information indicating a shortest navigation path to the goal in the web application relative to other navigation paths to the goal in the web application; and a cobrowse client configured to receive a document object model (DOM) of a current state of the web application from a cobrowse session for a web server hosting the web application, wherein the IVR platform is configured to: based on the DOM from the cobrowse client, determine a next navigation action along the shortest navigation path, and generate a user prompt for the user based on the next navigation action, and wherein the cobrowse client is configured to receive an updated DOM in response to user execution of the next navigation action.
 9. The system of claim 8, wherein the IVR platform is configured to: obtain a user cobrowse session identifier; transmit the user cobrowse session identifier to the cobrowse session; and receive the DOM of the current state in response to transmitting the user cobrowse session identifier.
 10. The system of claim 8, wherein the user prompt includes a user voice prompt, a user visual prompt, or combinations thereof.
 11. The system of claim 8, wherein the goal includes obtain an order status.
 12. The system of claim 8, wherein the shortest navigation path has the smallest total Euclidean distance of mouse travel relative to the other navigation paths.
 13. The system of claim 8, wherein the current state of the web application reflects a user interface current state from a user perspective.
 14. A non-transitory computer-readable medium storing processor-executable instructions, the instructions comprising: obtaining a computer-readable command based on user audio input, the computer-readable command being associated with a web application; determining a goal in the web application associated with the computer-readable command; obtaining information indicating a shortest navigation path to the goal in the web application relative to other navigation paths to the goal in the web application; receiving a document object model (DOM) of a current state of the web application from a cobrowse session for a web server hosting the web application; determining a next navigation action along the shortest navigation path; generating a user prompt based on the next navigation action; and receiving an updated DOM in response to user execution of the next navigation action.
 15. The non-transitory computer-readable medium of claim 14, wherein the instructions comprising: obtaining a user cobrowse session identifier; and transmitting the user cobrowse session identifier to the cobrowse session, wherein receiving the DOM of the current state includes receiving the DOM in response to transmitting the user cobrowse session identifier.
 16. The non-transitory computer-readable medium of claim 14 wherein obtaining information indicating the shortest navigation path includes determining the shortest navigation path among the other navigation paths to the goal based on a number of inputs required metric, a number of mouse movements metric, a number of keystrokes metric, a number of mouse click events metric, or combinations thereof.
 17. The non-transitory computer-readable medium of claim 14 wherein the user prompt includes a user voice prompt, a user visual prompt, or combinations thereof.
 18. The non-transitory computer-readable medium of claim 14 wherein the goal includes obtain an order status.
 19. The non-transitory computer-readable medium of claim 14 wherein the shortest navigation path has the smallest total Euclidean distance of mouse travel relative to the other navigation paths.
 20. The non-transitory computer-readable medium of claim 14 wherein the current state of the web application reflects a user interface current state from a user perspective. 